Guy Aridor (Northwestern University) - Evaluating The Impact of Privacy Regulation on E-Commerce ...

[Music] yeah thanks Tiffany um yeah thanks again for the organizers for uh inviting this paper and thanks again in advance for to Garrett for his uh comments at the end um so yeah so I'm going to talk about our paper evaluating the impact of privacy regulation on e-commerce firms evid evidence from Apple's app tracking transparency which will abbreviate his at through throughout a lot of the paper I'm Guy this is Joint work with young uch who I see is here on the call as well uh Brett Hollenbeck Maximillian Kaiser and Daniel McCarthy and so this I wanted to start with kind of I was like this uh infographic from The New York Times a few years ago so we're going to talk about uh sort of privacy regulation it's useful to talk about third-party tracking and so this infographic again we're going to focus on mobile apps this is on the web but kind of a similar Dynamic happens there where again you can see here's a user kind of going across a bunch of different websites here you see them on a inton post.com here's them on washingtonpost.com each of these little dots here is is some uh tracking resource um in these connecting dots are basically ATT tracking resource that kind of shares unique IDs right so when we're going to be thinking about third party data sharing you know this ID is going to allow for instance a platform like a meta to be able to see that user was on huffpost.com and went to washingtonpost.com as well and why is this a technology that's kind of been developed the main kind of use of this kind of thirdparty Data Tracking has largely been for targeted advertising right and so in particular this can allow two things the first thing is that it can allow firms to precisely Target using granular data on consumer online Behavior right so if I'm uh advertising on for instance a met or Facebook I might be able to Target people that use huffpost.com if I can kind of Link the identity of someone on Facebook to someone who visited huffpost.com and Beyond just uh allowing kind of an additional data point for targeting it also potentially allows for a virtuous cycle of targeting based on measurement right so in particular if I can track a user across a bunch of different websites I can also measure whether I I showed them an ad on Facebook and they ended up converting and again this can be used to optimize targeting now of course if you stared at this you may have thought you know from a consumer perspective it may be hard to kind of understand or even control kind of the ability to uh um for these firms to kind of collect this consumer data and again even if there was sort of no instrumental or or sort of payoff relevant reason for uh you to dislike this tracking you may sort of intrinsically dislike being tracked across a bunch of different websites and so this backdrop kind of privacy regulation has spun up over the last I mean it's been around for a while but it's kind of had a uh become much more prominent in the past few years where either governments or or industry players basically try to balance a consum privacy choices with cost to firms right so a lot of those in the audience today in in in tulo for instance maybe familiar with the general data protection regulation there's again uh numerous other types of these regulations that have spun up uh kind of inspired by the gdpr but there's also been industry initiatives right and so here we're going to fa focus on Apple's app tracking transparency again we're going to uh title this ATT throughout which is going to focus specifically on iOS Google also has a relatively large initiative uh called the Privacy sandbox which at least purportedly will be rolling out next year uh um but the quite of main point here is again uh both governments and uh industry players are kind of thinking about how to best balance kind of the trade-off between the costs to firms are the benefits to firms and the cost to Consumers now kind of a naive viewpoint would be to view this as kind of just being Zero Sum whereas we kind of take away a little bit from firms we potentially give some back to Consumers and so I want to think a little bit about kind of what's reportedly B the value of targeted advertising and in particular the value of targeted advertising on platforms like uh meta right and so purportedly they enabled the entry of kind of direct to Consumer firms right and so what are these types of firms you may be familiar with with some of the names on here these are kind of some of the original and and uh most prominent uh DTC firms like Casper bonobos Alberts Warby Parker and the main idea is that you know these are at least at the start kind of smaller firms that can sell exclusively on their own website and cheaply acquire consumers through targeted advertisement right and purportedly the kind of Rise of social media advertising and Target advertising online has enabled the success of these type of firms in 2022 uh these DTC sales for instance reached $155 billion and so in this paper we had a a conceptually relatively simple question which is privacy regulations but in particular Apple's app tracking transparency uh we wanted to quantify kind of what the economic costs of ATT were on these types of firms to try to understand again whether or not when we're thinking from the perspective of privacy regulation whether there's potentially Downstream uh harms to Consumers By changes in the composition of product markets okay so let me give like a high level overview of what we do in the paper and our main results and then I'll dive into all the details and so the first thing is again we're going to try to understand how does this privacy regulation affect advertisers and their bottom line right so the two types of data that we're going to have is we're going to get really granular data on how firms allocate their advertising online how it purportedly performs and importantly kind of their Downstream firm Revenue okay now the first question we're going to ask and again this is going to be uh particularly pertinent in the case of ATT is what is the magnitude of reduced advertising Effectiveness on platforms that exante were expected to be very effective and in particular impacted in particular ad platforms like meta okay and the first thing that we find is that again if we look at an event study of kind of like cost per conversion on meta there's a very sharp increase that exactly coincides with um the onset of attt now there's going to be some measurement issues that we're going to have to think carefully about uh and we I'll punt on the details of that so later in the talk but effectively what we're going to do is we're going to look within Advertiser at types of campaigns that are affected by ATT to those that aren't to kind of get a causal effect of kind of what's the reduction and Effectiveness here and we're going to focus on clickthrough rates and find that there's roughly a 37% reduction in clickthrough rate okay now the natural question is again we're looking at this policy in the wild and so if ad platforms like meta in particular are impacted the natural question is how do advertisers kind of respond to this right and so what we're going to do is we're going to look just at a very simple event study to show that there's some moderate reallocation from The Meta ad ecosystem to the Google ad ecosystem okay and then finally kind of the core question we want to look at is we really care about kind of the downstream impact on firms side costs here okay and so we care about what are the magnitude of Revenue reductions and in particular we're firms that kind of like we more dependent on meta advertising which again we uh L link to kind of these direct to Consumer firms were they able to kind of mitigate this shock and largely what we find is we use measures of uh preat dependence on meta and iOS and find that large Lely these types of firms were unable to kind of mitigate the shock leading to a kind of similar similar 30 37% reduction in Revenue which is again a quite sizable uh reduction in revenue and then we look at heterogen 8 and we find that this 37% is largely coming from these sort of smaller direct to Consumer firms um which again kind of puts into question you know to what extent does this regulation kind of harm the uh affect the ability of these types of firms to prosper and then finally we have something in the Revenue data that's going to allow us to kind of Link this back uh to show that the mechanism is coming from reduced new customer acquisition again because of the inability to kind of Target advertisements and so the kind of key punchline takeaway from our paper is that these direct to Consumer type of business models are threatened by ATT which again I think puts some ambiguity into kind of like how do we think about the welfare effects of these types of regulations for consumers okay so that's the broad overview going to get into the full details in the rest of the paper I think first before going into those details I want to briefly talk about some uh related work and so kind of within the space of privacy regulation uh papers I think kind of the most relevant are those that sort of The Economic Consequences of at and so the two most relevant are uh paper by wernerfelt at do which which runs kind of field experiments on Facebook uh experimentally removing uh third party data and again relative to this paper kind of we're looking here at kind of the equilibrium effects of at where we're thinking about how both met is adapting how advertisers are adapting and kind of measuring to what extent is uh or firms harmed after they can potentially mitigate the shock uh CIS and L mer also kind of look at reduced ad Effectiveness and again here we're also going to look at uh Downstream outcomes on firms uh there's a number of of papers that kind of look at app responses including Tiffany's paper uh here again we're focusing on advertisers and not applications per se now outside of you know there's been a number of other papers kind of looking at other types of privacy regulation and again hopefully we're getting to the point in a literature where again we have all these types of natural experiments generated from privacy regulation we're slowly learning kind of what are the economic costs of this and hopefully uh can allow us to kind of better craft privacy regulation in the future I think the most relevant of these other privacy regulations is the the gdpr in particular because it has a uh a sort of enables consumers to kind of have op- and consent um and again I'm not going to go into there's a a number of really good papers on this topic Garrett has a really nice review on this I think the two most relevant are the the um the one that Yong and I worked on with tobas and Garrett's paper with with Sam Goldberg and Scott Shriver we're going they kind of look at uh the downstream effect on on e-commerce firms I think relative to gdpr one thing that's really nice about at is one the main provision is really this kind of opt-in data sharing but two there's kind of a much cleaner optim into implementation because in gdpr there was a lot of heterogen and kind of extent to which firms were complying with the regulation the types of opt-in prompts here as I'll show you there's kind of a very high compliance um and so it's a clean measure a clean policy to kind of look at the cost of opin privacy regulation and the second is that because it specifically impacts only iOS we can kind of use it as some variation to say something in particular about kind of the value of meta advertising and again I'm a little bit vas here potentially but we have a a journal of economic literature piece coming out later this year which kind of identified that kind of thinking about the value of the social media ads on Downstream product markets is something that's kind of we don't understand as much about but seems pretty important okay and so with that let me dive into the the details of the paper so the first thing is what is this app tracking transparency and so if you have an iPhone you've most likely come across this right so after you update to iOS 14.5 if you go to an application uh this popup comes up where it says again this is kind of standardized across all the different applications you see on the phone um you know allow this app to track your activity you can ask the app not to track or you can allow right and so industry studies that kind of look at the extent to which consumers actually say allow or not allow so the op out right here is pretty substantial right so it's nearly 80 80 to 85% according to the IND industry sources we cite in the paper and so this is pretty sizable right now what does this actually do right and so a typical use case in particular for instance on a platform like meta is what will happen is an Advertiser will buy an ad on Facebook Instagram or the sort of meta Audience Network what's going to happen is that you know an Advertiser typically wants to kind of measure you know a success for an advertisement right so what they typically do is to put some pixel on their website or their application that whenever a consumer kind of makes it to that page it's basically going to send an HTTP request back to meta and tell them hey here's the idea of the person that just converted and meta can then use that to measure whether or not an ad worked again and uh optimize distribution right so when we're thinking about targeting on a platform like meta you know it's not just kind of like I put in my targeting information but kind of this delivery optimization also plays a big role and so the main consequence of at is basically this third part you can't this kind of makes it so that the firm can't share back What's called the idfa back to the end advertis or back to meta and so what's the what's the implications of this for advertisers and so this was expected to kind of disproportionately impact ad platforms such as meta uh in particular again iOS heavy platforms third party optimized campaigns and the result is kind of worse uh targeting in measurement advertisements right first you know again Advertiser and meta kind of can't link B ads back to conversions really impacts the performance of delivery op optimization but also how has less behavioral data now one thing that I think is important to keep in mind as we're going through the paper is that kind of meta after the policy is trying to do a lot of things to kind of potentially adapt to this right so in in essence you can kind of think of a lot of the things that they're trying to do is to move from kind of a deterministic attribution model where again I see the ground Truth for whether or not to someone converts to a more probabilistic attribution model kind of through this aggregated event measurement and again the main idea is to kind of like have some model of conversions for the set of consumers that opt out right So Meta is doing some type of adaptation uh here okay so that's the broad overview for the policy again impacts all the apps on the iOS store uh doesn't impact anything off off of iOS okay now what are the the data sources that we're going to rely on this paper and so we're going to rely on kind of two pretty complimentary data source that kind of in Union are going to give us ad spending and Revenue new data separately for for 4200 Global firms okay and so the first re uh resource we're going to use is uh data source a which again uh so I'll refer to as data source a later but you can kind of think of it as the data set that we use primarily for Revenue which comes from a firm called grips intelligence and so what what kind of data are we seeing here and so it's roughly aggregated first-party Google analytics data from direct data Partnerships right so a firm will kind of op in to allowing grip's analytics to view uh its direct Google analytics data and so what we're going to mainly use this data set 4 is we're going to observe over time transactions and transaction value by a few things and so the first thing which is going to be important for our measure or for our analysis is we're going to have some measure of referral source right so this is going to be coming from some last touch attribution or basically for instance if I was on facebook.com and then I went to let's say nike.com And converted this would attribute uh kind of a um a conversion back to Facebook importantly we also observe device type and operating system right so we're going to be able to see for instance uh if a consumer what fraction of purchases come from iOS versus Android okay now in terms of panel composition the grips intelligence data is going to cover a wide range of firms and in particular we're going to be able to track you know nearly 116 billion in yearly Revenue through this data set and so important for our analysis it's not just going to contain kind of exclusively direct to Consumer firms but also uh many sort of Reta broader retailers that kind of again also have brick and mortar stores and such and so when we're going to be doing our analysis we're not necessarily going to be comparing kind of within DTC firms but our treatment group will effectively have DTC firms in the treatment group with a more non- DTC firms in the in the non-treatment okay and so this is the first status that we're going to have that's going to give us a a broad meas measure of Revenue the second is going to come from an anonymous advertising analytics firm that's going to provide us aggregated data on spend performance on meta Google and Tik Tok and so what we're going to see is again uh weekly spend number of Impressions across campaigns number of clicks across campaigns and then the number of conversions that's kind of coming via the sort of uh pixel right so this kind of gives us a view for from the perspective of a meta or a Google or a Tik Tok like how many conversions are they actually observing after the at event and we're going to see some breakdown of this so for instance for meta we observe the campaign objectives which we'll use in our analysis for Google we can see whether the spend was on search display Etc okay now from this data source B we're going to have a subset of the data that's basically going to give us direct sales uh we're going to have monthly revenue and importantly we're going to also be able to kind of have a point of sale identifier that's kind of like an email or a phone number that's going to allow us to kind of again break down by new repeat cons customers right so we see the kind of end aggregation of what fraction of consumers were newer repeat con consumers okay and again just to kind of show you some broad summary stats of this um so again you see here the data sets have uh pretty pretty decent variation both data sets contain a lot of these smaller director consumer type of firms uh data set b skews a little bit more towards smaller firms but data set a uh kind of uh yeah data set a has a broader range of firms uh we're going to have variation in both iOS and metad dependence in the data set a which we're going to exploit in our uh analysis okay and so that's the um sort of policy details and data sources does anyone have any broad questions about the policy or the data before I start to get into the results or okay I will I will keep going then so the first thing I we want to look at here is we want to measure to what extent our meta ads in particular impacted okay and so I'm just going to show you some event studies and what are these two things right so this thing up here is the CPP which is basically just the average spend per pixel attributed conversion right so this is you know again given the a firing of this metap pixel what's the average amount of spend that has to go into kind of uh for each those pixel conversions and then the second down here is again an event study that looks at the log number of conversions right so this is again meta observed conversions um and how does this track over time and so if you stare at these two things there's kind of a clear Point here where after at you know there's a a gradual increase up here and then there's nearly a discontinuous increase in CPP you see a similar pattern for conversions um and so again it's pretty suggestive of a pretty large impact here right so so this is roughly kind of a 50% increase in the cost per pixel attributed conversion now a natural question as you look at this it's pretty suggestive that it's coming from attt but you know can we do anything to kind of make us a little bit more confident in this and so what I have here is so this is the same CPP plot and one thing which is somewhat nice about a policy like at is that basically whether or not again the end consumer kind of uh sees the opt-in prompt really depends on kind of them updating their devices right and so there's some uh slow adoption of kind of like whether or not people actually update to iOS 14.5 at some point in early June kind of apple pushes uh to kind of have a lot of people uh update their device and so this red line here you can see kind of the the right side of the y- axis here is kind of the fraction of devices updated and one thing which is kind of cool is again you can kind of see here this is uh obviously there's no no real adoption of HT before the there's after April 25th 2021 which is when the ATT comes into place you see a gradual adoption which again almost exactly coincides with kind of the increase in in CPP and then whenever there's this big push to kind of get a lot of people on iest 14.5 that's precisely the week in which you see this kind of spike in uh the the sort of CPP right and so you see that the increase in cppp again precisely coincides with the adoption curve of iOS 14.5 now this is pretty suggestive that the effects are coming from at but still this measure on the left hand side has some measurement problems right and so H induces measurement issues both for meta and for us as the econometrician right because we again only seeing what what meta seeing and so what we're going to do is we're going to take uh inspiration from kind of this Warner F all paper and kind of look at it observational analog to kind of what they do in their experiment where what we're going to want to do is again because we observe this breakdown of campaign objective within meta we can look at conversion optimized campaigns which again are those that rely on this uh thirdparty pixel to kind of do attribution and measurement um to click optimize campaigns which kind of all contain within the meal ecosystem and the nice thing about this is it this kind of analysis is going to allow us to control for differences across advertisers right because we're all looking at within Advertiser comparisons and again is also going to potentially control for potentially meta adaptation over time right met is making changes to its uh to its uh targeting algorithm okay and so in particular the empirical specification we're going to do is kind of a standard two-way fixed effect with an Advertiser difference and difference is and we're going to largely look at kind of two dependent variables here the first are kind of success rates right how often does the campaign achieve its objective right and so in particular in the case of uh conversion optimize this is going to be off-platform conversions in the case of Click optimized this is going to be uh clicks on the platform and again this first dependent variable is still subject to these same measurement issues the second variable is going of completely on platform right so there's going to be no uh potential measurement issue here now when we're thinking about kind of like which advertisers to estimate the specification over one thing we have to be a little bit tricky about right is one there might be a concern that there's some reallocation here um from offat form to on platform and so again I won't go in the details here but we don't observe kind of too much substitution on on the kind of uh of of advertisers switching from kind of off platform to on platform we see a little bit of of movement on the ex extensive margin but largely advertisers on meta kind of continue to predominantly use kind of off-platform conversions uh even post at and again anecdotally this was so because kind of you know Meta Meta told people this was the most effective way to use the platform if they cared about off platform conversions and so what we're going to do is we're going to look at a set of advertisers who had any spend on both objectives before at and kind of compare performance across time okay and so what is what is it that we find and so again the top panel here is this difference and differences for success rates the bottom panel here is the difference and differences for click-through rates and again what you see is is similar to what we saw before which is you know there's seems to be a beginning of an effect the the first two months but then after ATT really kicks in and a lot of consumers adopt you seem to see three or four months out a a relatively steady kind of reduction in success rates and clickthrough rates again with some little variation coming from the holiday season um but in general kind of again if you Benchmark this to kind of the average uh clickthrough rate in the Baseline period the the key takeaway is that kind of there's a 37 produ per reduction in the click theough rate for conversion optimize s right and so that increase that we saw before for from uh the sort of increase in acquisition cost uh is not necess is is not just a measurement issue but there's actually a big reduction in effectiveness of ads odds on on ad Effectiveness on meta okay now this leads to the Natural question which is given reduced eff IND of AD of meta ads what did advertisers do and so I think kind of obviously this is going to be relevant for thinking about this question of at but another interpretation of kind of the at shock is the kind of view it as an exogenous reduction in kind of the quality of meta ads right and so in some sense you can think about the kind of substitution exercise and whether or not it impacts Downstream revenues as potentially also being useful for antitrust debates around meta because it's going to teach us something about to what extent was uh kind of meta advertising substitutable for a lot of these advertisers uh preat or post at okay and so you we're just going to have some some some pretty descriptive results here and so what are we going to do so again we observe spend on meta Google and Tik Tok we're going to focus just on spend primarily because uh kind of the quantity variable here varies across ad platforms on Google you may be optimizing for clicks on Google search on meta you may be uh or sorry uh paying for clicks on Google search whereas on meta you may be paying for Impressions and so we're just going to focus on spend primarily and we're going to conduct two exercises right the first is kind of uh event study for relative spend shares across these different platforms and the second is again more differ differences for more metad dependent advertisers and so what is it that we find and so again in our in our sample here kind of the basine uh prevailance of of Facebook ads was with a mean market share of 73% in the pre period uh and so what you see here you see that before ATT happens there's not much change in kind of the relative spend share on Google or Facebook or meta um and you kind of again after ATT see a pretty Stark reduction in overall spend on meta and a gradual increase that's kind of offsetting this to some extent on on Google right and so in terms of uh sort of relative reallocation this kind of reduces the market share of meta Ben share by around 10% within our data set okay um and then the second thing we can do here is again we can kind of look at our metad dependent advertisers more likely to substitute and so again we can look at kind of an across Advertiser here difference and differences where we look at the relative spend impression and clickshare across these different platforms and again it's suggestive that kind of more metad dependent advertisers are likely to substitute it somewhat suggest more substitution to Google display than search which again is relatively uh intuitive because Google display is kind of the behavioral targeted advertising equivalent on Google and so again we're seeing some some reallocation here okay okay sorry I just have a um like a question um yeah so just like would you be able to tell the spending magnitude instead of share is there any way to think about that uh yes so we can look at this the spending magnitude again we get into details in the paper there's a lot of weird stuff going on with at in terms of prices and changes in quality um and so we just focus on spend share here because we're kind of interested in sort of are is there a relative reallocation across these different platforms uh to give some sense for this agree that kind of like one could do a full-blown exercise of kind of measuring substitution patterns in a more disciplined way but again that exercise is uh complic very complicated in its own right and so here we're just looking at some descriptive results okay thanks okay okay and so now again we've shown a reduction in effectiveness of meta ads we've SE again some descriptive evidence for reallocation and now we want to get into kind of the meat of the of the question which is like did this impact Downstream firm outcomes and in particular The Firm revenues okay and so were firms able to mitigate the shock in this Effectiveness and again there's kind of two interpretations here that I think are are relevant for guiding kind of like how we want to think about this empirical exercise right so the main variable of interest here is going to be Downstream firm revenue and again the first interpretation is if we primarily care about the impact on sort of DTC firms which are inherently more metad dependent we kind of want to look at sort of some measure of of preat metad dependence the second interpretation is the again you of we can view this exercise as you know being an exogenous shock to the quality of meta ads which is going to give us some kind of revealed value for how was meta how valuable was meta advertising for these types of firms before ATT okay and so our empirical strategy is going to be to stratify advertisers by preat dependence on meta okay and so what we're going to do um is again we're going to have a balance panel of firms here that are in the data set but in particular we're going to use this measure that we have from data set a which is the uh share of attributed Revenue coming from meta and we're basically going to stratify serms firms kind of preat based on this relative share okay and so we're going to define the share again from the year before meta and Define kind of treated firms here as being firms that kind of are above the median uh share of attributed revenue from meta to firms that are kind of below the median share and again we'll have some alternative specifications using iOS but this is going to be kind of our main specification here okay and again here we're going to again be doing a two-way fixed effect difference and differences uh but again here we're kind of doing a cross Advertiser as opposed to before within and again we're going to Cluster our standard areas at the Domain level okay and so the first thing I want to look at is just plotting plotting the data over time right so this is again the the dark line the sort of non dash line here is the control group the dash line here is the treated group and again this is just plotting means a log you mean a revenue over time and again you kind of see that the control group and the treated group match each other nearly perfectly until the album set of at at which point they diverge and you see that again the treated firms kind of have a reduction in uh in their overall Revenue we then take this again to kind of the difference and difference specification uh kind of looking at time varying treatment effects here and so again what do you what do you see here the first is that there's there's no real evidence for pre-trends before ATT and the second is that again kind of consistent with what we saw before there seems to be some in uh beginning of an effect the first couple months after ATT and then roughly after four months again you kind of see a flat lining right so once kind of consumers have all adopted ATT um there seems to be a pretty persistent uh negative effect on Revenue here right and so again this is despite the fact that we know that meta was doing uh kind of adaptation in terms of trying to move towards this more probabilistic attribution models this is despite the the sort of reallocation we see of some of these firms towards uh spending on Google right this kind of indicates that these firms were kind of unable to really substitute for their preat dependence on meta advertising okay and so what are these the magnitudes of these effect sizes and so again we can then turn to kind of like the sort of standard difference and difference estimates here and so again the the main two-way fixed effects estimates kind of leads to a roughly 37% reduction in in revenues right again which is which is quite sizable right you think about you know a 37% reduction in in firm revenue is you know big enough to kind of potentially deter entry from firms in the future the we can do a bunch of kind of alternative specifications for this to kind of show robustness again one concern with our empirical analysis may be that some sort of categories are potentially more likely to be dependent on meta and so again we can kind of control for category month fixed effects to to to account for this and again we get nearly the same uh magnitude of of of the effects we can look from 0 to three months and four plus months and again the effect size is bigger if we kind of condition on um the months when we kind of know that there was already full adaptation of from the perspective of uh or there's full adoption of at perhaps the starish result here is we can we can do another specification where we stratify where we kind of uh consider only the subsample of kind of smaller firms which again we're going to Define here as below median uh Revenue in our data set and above median data set we'll call large firms and we find that most of this effect largely seems to be coming from these smaller firms right so this this uh 30 7% which seems pretty big is potentially even higher for these smaller firms um which again is is a a little bit a little bit troubling in terms of the costs of this type of Regulation um and then finally we can look at whether or not kind of the overall number of transactions fell uh which uh we find that roughly there's a 24% reduction in transactions okay and so again this this pretty pretty sizable effect sizes uh we can look at kind of you know another measure of this right you may say okay well more natural me meure of kind of Crea dependence is the degree to which a firm is dependent on iOS sales right and we have some discussions in the paper and you know we we kind of uh this measure is correlated but not perfectly correlated with meta with meta usage and so this is capturing a potentially different measure of dependence and again uh you know I'm just showing you the time varying plot here but you can look in the paper for the full tables we basically find like nearly identical results when we look at kind of iOS dependence as opposed to to metad dependence okay um which again kind of seems to indicate that the the effect sizes here are pretty large and pretty robust across uh reasonable ways of kind of doing this analysis okay now the natural question may be okay so we've kind of independently shown advertising Effectiveness seem to went go down on meta revenues seem to go down on more metad dependent firms uh we can kind of potentially tie this Loop by kind of looking at the relative share of kind new versions repeat orders right so if it's really kind of this reduced Customer Effective uh consumer acquisition Channel That's the driver of this revenue losses we should see that you know um there's a reduction primarily in new customer orders and so here we're kind of switching to use our our data set B which has this measure of kind of new vers returning customers and the first thing we can do is again we have a measure of kind of like what is the fraction of orders that are uh from New Verse repeat customers and again using this a similar type of specification we find that this is this corresponds to roughly kind of a 10% increase in the sort of repeat order ratio right um then we can break this out based on kind of like how many new customer orders were there how many repeat customer orders were there and we find uh pretty strong and again comparable effect sizes to kind of the reduction in new customer orders for kind of more metad dependent firms we don't seem to see kind of any potential offsetting coming from from repeat customers right so one alternative Theory may have been that sort of these firms can put additional money in kind of retaining and increasing the order value for additional consumers again we don't seem to see much evidence of this but we also again seem to find that the revenue impact is largely driven by the loss of new customer acquisition here okay um and then again we have some additional robust for this Revenue results using the Shopify data we can control for changes in firms side costs use different measures of dependence and we find similar types of effects okay okay so I think I'm nearly out of time so let me finish up here by again uh sort of concluding what we what we find in this paper and so I think again uh our paper kind of contributes to this debate around the economic costs of privacy regulation again at is a particularly nice policy for measuring the effect of opt-in privacy regulation given the transparency to Consumers the high opt-in rates we can really uh measure the effects here what are our main empirical results again we find a 37 production per reduction in these click-through rates for conversion optimized meta ads we see some reallocation from Meda to Google despite this reallocation we find that kind of more metad dependent firms have a pretty large reduction in revenue of nearly 37% and this largely Herms small firms what are the kind of again key policy and managerial takeaways from this and so I think again the magnitude of the effect sizes in particular for smaller firms kind of points the ATT potentially threatening the viability of a lot of these direct to Consumer business models and so again I think this kind of opens up the question of you know when we're thinking about the sort of consumer welfare benefits this regulation like undoubtedly consumers are getting a lot of benefits from kind of being able to additionally protect their privacy I think one thing which our paper points to is that it's potentially also going to be important to understand the sort of consequences on Downstream product markets and how that affects consumer welfare to kind of do a full welfare analysis of these type of Poli policies the second is that again if we view uh sort of at as being an exogenous shock to the effectiveness of meta ads and seeing what that teaches us about um kind of its value and the degree to which advertisers can substitute I think again given the magnitude of the revenue reductions I think it's hard to argue that the preat value of meta ads for these firms was high but the fact that they were unable to kind of adapt to this also does suggest that there are potentially limited viable substitutes for these firms right and so we think this provides some useful evidence for the antitrust debates around meta um okay and so with that I will kick it over to to Garrett uh thanks in advance Garrett let me yeah so thanks sky for the presentation so uh Gary please uh go ahead with your discussion sounds good well thank you for the opportunity to discuss what I think is a really outstanding paper uh ATT is perhaps the most consequential restriction on Modern identity B based digital advertising uh but at the time the public and academic discussion did not seem to really appreciate this point um and the first papers on this literature focus on the consequences for apps uh which is important uh but the impact on advertisers is a critical part of the story as well uh now of course it's one thing to say that it's another to obtain really good data and the authors obtain multiple useful data sets that uh clarify this question and so they have a very nice contribution which is to show the value of digital advertising and specifically identity-based advertising to the economy um showing that advertising plays a critical role in growing firm revenue and in fostering competition so my only complaint really about this paper is that I wish I would have uh I wish I would would have ridden it um so a couple kind of overarching comments the the first is to be less bashful about the policy takeaways uh so yesterday the news came out that the European commission is going to prosecute uh met under the digital markets act for offering consentor pay but not offering this third option with this so-called contextual advertising um and this is part of a larger move by the European commission um that seems to be very suspicious of targeted advertising identity-based advertising and I think the message of this paper is that whatever privacy benefits these policies provide they have a substantial cost to firms and a substantial cost to competition and we should uh meet ABB thinking critically about these regulations um I think they also should be framing the contribution as informing the ongoing antitrust investigation against Apple so um Apple unilaterally imposed the at policy on its platform um and notably as part of this it characterized its competitor's use of AD identifiers as tracking uh but its own use of AD identifiers as personalization uh and while hampering the ad effects of or the ad efforts of his competitors Apple was able to grow its own ad Revenue uh to the tune of 238% uh so its search ads program grew by $3.7 billion do in 2021 and as a result of this there's multiple antitrust investigations uh of Apple so France issued a statement of objection in 20123 uh there's four French online advertising groups that actually sued um Apple for an antitrust complaint in 2020 uh the US is investigating Apple's mobile platforms which may include ATT uh and the CMA the British competition Authority promly discussed ATT in its mobile ecosystems Market study so I say all this because uh the paper's findings are material for calculating potential damages or fines uh and I think the authors could could play that up a bit and then my second major comment is that I like the authors to do a better job of placing their findings in context uh so the estimated treatment effects are really high at almost 40% for Revenue so um I just want more context to think about this so I think one critical piece of data that would be very easy for the authors to provide is what is the share of Revenue attributable to meta into iOS in the pre period for the treatment versus the control groups and we see in the summary statistics that the fifth and 95th percentile for meta ranges from 0 to 18% which is quite low relative to those effects uh for iOS share it's between 3 and 51% uh which be more commen with those effects um admittedly this this doesn't give you the whole picture because uh we know that uh the these uh Shares are on a last click attribution basis and we know that the um relationship between incrementality um and last click attribution is not one to one albeit we do have some great research from Brett Gordon uh guy aridor his colleague at Kellogg uh quantifying this relationship across many different experiments um I also would be interested to see the effect um computed by uh device and uh um an operating system to compute iOS specific effects um admittedly the authors are interested in this kind of General equilibrium you know post- adjustment effect which I think is all well and good U my interest in the iOS specific partial effect would be as more of a validation check because really that is where we should see the largest effect of at TNT and so it would be nice to to break that apart so um again really really great paper uh very important contribution to literature into a very important and active policy discussion and I commend the authors for uh really excellent work on this sub subject