Agenda Request - Measurement use-cases deep dive and requirements #51
Comments
csharrison
added
the
agenda+
|
Should we start with a review of the fundamentals of attribution measurement or are we confident that everyone already understands them? |
I think it depends on how much time we end up allotted to this topic. If we have multiple hours, an overview is good. Otherwise my preference would be to maybe assign some pre-reading and get into the deeper topics quicker. |
|
I think this is a good idea - I would also like to put forward the incrementality / lift measurement use case if this has not already been discussed. |
|
IMO we should allocate one entire meeting to this topic. It's critical we
start to converge on some technical ways forward and that starts with
agreement on the problem we want to solve.
…On Tue, May 10, 2022 at 9:37 AM Daniel Smedley ***@***.***> wrote:
I think this is a good idea - I would also like to put forward the
incrementality / lift measurement use case if this has not already been
discussed.
—
Reply to this email directly, view it on GitHub
<#51 (comment)>, or
unsubscribe
<https://github.com/notifications/unsubscribe-auth/AAIPLILWBELOBQWONXSY6A3VJKGGJANCNFSM5VOODGMQ>
.
You are receiving this because you are subscribed to this thread.Message
ID: ***@***.***>
|
|
+1 to @ekr. |
|
I'm all for spending an entire meeting on this topic, but it would be helpful if we identified a few sub-topics within this to divide the time, and make sure that we use those 6 hours productively. A few sub-topics that have been raised above:
I'm not sure we actually want to cover all this, and this list probably misses some other valuable sub-topics. I do believe it's important to agree on some of these for the agenda before next week, though. |
|
I'm with Charlie on avoiding the basics. We should be beyond the basics at this point. Though I doubt we'll get there after just one session, I'd like more of a shared understanding of the requirements so that we can more easily determine whether a given solution is useful or acceptable. Just to pick on one of these, I would like a better understanding of the interplay between different optimization cases: bid strategy optimization, creative optimization, targeting optimization, and content optimization being obvious aspects. In all these cases, I'd like to get a better idea of the extent to which - if at all - optimization needs more data than reporting, as opposed to having a need for different data. The answer for a lot of these use cases - though maybe optimization more than others - is probably just "moar data" at the first N levels of simplification. What I'm looking for is a commitment to try to keep looking for more creative answers than that, because that answer isn't helpful in terms of getting to a solution that is respectful of individual privacy. Erik's suggested topics seem like a good way to start. |
csharrison
changed the title
|
It would be great if somebody could volunteer to lead us through this discussion. |
|
If we want to work through the list I proposed above, it might be useful to have different people lead each of the 3 subsections. I'd be happy to lead the discussion around item "1. Measurement use cases." |
|
I have a few slides prepared on optimization that I can present. I'd also like to discuss the 3p reporting use case. Otherwise I'm happy to have you lead Erik. I can volunteer to lead discussion for 2 and 3 though I don't have as much explicitly prepared. |
|
It would be great if we can include here also discussion about the reach and frequency measurement application, which Raimundo presented last time, but getting a bit deeper on its requirements. |
|
Hi everyone. When we were discussing @eriktaubeneck's list internally, we weren't sure the intended distinction between:
Am I right in assuming both are intended to be about figuring out how the listed measures drive some subsequent desired (digital) outcomes? And, given the first is separate to the second, does this mean the second is about attribution considering a range of touchpoints (multitouch attribution), or is it more intended to be an augmentation of the first by allowing for other types of touchpoint to be incorporated? (As in event-level ARA as currently proposed/prototyped in Chrome.) For what it's worth, I generally consider views a (proxy) measure of impressions, while clicks are a (proxy) measure of outcomes, but this means clicks kind of straddle the line between media activity (something running on a media owner's property [1]) and (digital) outcomes on an advertiser's. (They are after all an approximation to loading the thing that is clicked through to, which might be a webpage, an Android Activity, or whatever.) (By the way, I don't really think of either click or broader impression attribution as being at the level of "use cases", because they're both means to gaining some understanding of how media activity is affecting outcomes. That itself is a group of techniques serving a range of use cases from in-flight optimisation to future campaign planning. I don't want to be the annoying person who asks us to define the concept of use cases carefully, but it'd be helpful to keep an eye on how all these things are actually useful for advertisers, publishers, and their agents/processors/service providers.) When we talk about measurement internally, we also include "basic" reporting of media activity (views, clicks, CTR and then getting into reporting measures around things like viewability & bot fraud, and then calculated measures such as vCPM). Since we're including reach & frequency, which is closer to basic reporting than attribution because it doesn't require outcomes, I thought I'd throw that out there even though it's not particularly connected to attribution reporting. [1] although acknowledging that you can also attribute from activity within your own properties. |
|
If there is a broader category of metrics where R/F falls into, maybe we can have a separate agenda item. This also relates to this proposal patcg/proposals#13 |
|
Yes I think it might be better to split R/F into a 'campaign metrics' category rather than measurement, including the "basic" reporting of media activity (views, clicks, CTR, viewability etc.) James mentioned in his comment. For this discussion it would be good to focus on pure measurement use cases only i.e. based on outcomes (and attributing those outcomes back to the campaign) |
|
The primary consideration here is cross site activity, which currently (or previously, for some user-agents) rely on third party cookies / device identifiers. Attribution - typically formalized as joining a source event on one site a trigger event on another site - is a clear instance of cross site activity. The reason I separate out "click attribution measurement" and "impression (click, view, etc) attribution measurement" is just reflecting the current proposed APIs treat this differently. Private click measurement only supports attributing trigger events after a click, and the event level attribution API allows for more entropy to be included post click than post view. Reach and frequency is also a use case that is worth considering here, as the goal is to count the unique number people to which impressions are shown, cross site (and often cross platform.) Are there use cases for "basic reporting" that require cross site activity? It seems like the ones listed (views, clicks, CTR, viewability etc.) could be handled within the first party context. |
|
Thanks Erik, that's a helpful reminder of where we're focussing right now :) The only "basic" metric input that currently relies on cross-site information / digital identifiers that I'm aware of is the way some vendors infer "human-ness", but that's more about the detection than anything to do with measurement (at least the way it's done now). I don't think that's worth folding in here. |
|
It sounds like there is enough here for more than an hour of conversation, a lot of topics and back and forth in just this thread. I'm going to reserve an hour on day one, but also place time in for open discussion on measurement topics that this can overflow into on Day 2. |
|
If we only have an hour, my preference would be to focus day 1 on the optimization use-case, and use day 2 for spillover topics. Does that sound OK to people? |
|
@csharrison I agree that we should focus the conversation, given there is only 1 hour. However, I think the core question at hand right now is: "What is the first use case that the Working Group should work to standardize?" We have been working in the general direction of some form of cross-site attribution measurement as that first use case (something in the area of Private Click Measurement / Attribution Reporting API / Interoperable Private Attribution,) but I don't think we yet have formal consensus on that use case, nor do we have the specific details of the functionality that the use case would need to support. Open questions I'm still aware of:
I'm doubt we will be able to reach consensus on all of these in the meeting, but it does seem prudent that we reach consensus on the use case the Working Group should tackle, find members to participate in that Working Group, and task them with the right questions to formulate an opinion (or options) on. I don't mean to say we need to solve "Attribution Measurement" before we begin talking about the optimization use case, but I do believe that it's important for the group to unblock and begin more formal work on that use case (or decide that it's not the right first use case) before we begin work on others. Once we have consensus on that first use case, and the Working Group can work on it in parallel, this group can consider additional use cases. Also, the optimization use-case seems to rely on attribution (at least in the formulations I've seen.) |
|
+1 to full @eriktaubeneck post above and…
…🙌🏼 We would be well served to start thinking in terms of building blocks like this. While opt also has real time signals, the feedback loop we stand to create here is foundational. |
|
Hey @eriktaubeneck, just to be clear I was considering the optimization use-case to be one specific use-case solved by attribution measurement, just one that is not as well-understood by participants. That was my intention with prioritizing the conversation, to make sure this isn't missed when we start diving into details. Before we discuss privacy definitions, technical details (budget grains, etc) I think it would be good to have a conversation about the high level use-cases we want solved here. I consider optimization to be one of those, alongside use-cases like reporting. |
|
It will be great if we can have sometime in the overflow for applications that do not need attribution but currently rely on 3p data such as reach and frequency. I think these types of applications may present simpler functionalities to be solved without the attribution part. But they also present another interesting question about interoperability between the browser APIs and other APIs such as Cross-Media Measurement. |
|
@csharrison that makes sense, and I agree this is a use case we will want to work towards support. That said, there is obviously a big tradeoff between a system which supports a bunch of the sub-use cases, and working towards a first standard that has more limited scope. I think the most important outcome from this meeting is getting consensus on the first use case we want to solve for, and establishing that as the starting effort for the Working Group. I know I've heard calls for starting with a first standard that is somewhat limited in scope, so that we can make progress and build momentum towards solving more of the sub-use cases within measurement, as well as entirely other use cases. I do think it's important to understand all the sub-use cases (e.g. attribution reporting, reach and frequency reporting, optimization, etc), and to consider the compatibility of those within the standard that ends up being developed. It would be great if we build a standard for one sub-use case, but with an approach that would be reasonably compatible for other sub-use case. For example, I would say that both the Aggregate Attribution Reporting API and IPA, if standardized for reporting, could be done in optimization compatible way. |
|
@eriktaubeneck I'm fully supportive of this group's approach of starting with a more limited first standardization effort. But the prerequisite is being confident that we are picking the right limited-scope API to work on. I believe there is widespread agreement on the ad tech side that the "optimization" (ML training) use case is important. But on the browser engineering side, this group has only spent 10 minutes on what that use case actually is. That means we're not even in a position to answer fundamental questions like "Are we trying to design an API that can support both the measurement and optimization in the future, or would those be two different APIs?" We don't want to wait until after we've painted ourselves into a corner to ask the question that helps us pick which corner we should paint towards. Yes on scoping down, but in a fully informed way. |
|
@michaelkleber - I don't disagree with any of that. However, looking at the original conversation on #50, the impetus for this agenda item stems from @ekr's comment:
It don't understand how we get from that to spending the entire hour on the optimization use case. I'm am in support of talking about the optimization use case, but I want to make sure that we also make progress on building some consensus on what the Working Group should be producing.
I agree with this sentiment, but wouldn't this be more in relation to a specific solution for a use case, rather than scoping the use case of the first standardization effort? Reporting generally seems like a prerequisite for the optimization use case, so I don't see the attempt to reach consensus on reporting as the first sub-use case to tackle as painting us into a corner. We could then use the next meeting (as well as Github) to try and reach consensus around the other important sub-use cases that should be in consideration of the standard. |
|
I definitely think it's very important to get the use cases nailed
My best case scenario for this meeting would be:
In terms of meeting time, I think it would be good to have
As I think I've said earlier, I would actually vote to punt everything |
I would prefer to lump the optimization use-case with the exhaustive description of the conversion measurement use-case, then discuss the gaps and next steps. This would basically be @eriktaubeneck 's section (1) from above, removing the reach / frequency use case for not being directly related to conversion attribution (and for sake of time). We have 1 hour 40 minutes. What do we think about spending the first meeting (1 hr) on use-cases (both reporting and optimization), then the overflow meeting (40 min) for gap analysis and next steps? |
|
@csharrison Please do share the slides as soon as you can. |
|
Slides are here: I apologize for not having them ready beforehand. |
|
Thanks for putting those together and driving the discussion, @csharrison! |
|
@csharrison thanks for sharing. I have also loaded them into the meeting repo. |
|
Thanks @seanturner . I might add a slide before tonight in which case I'll update the PDF |
|
In the discussion, there was interest in having some buy-side voices on measurement and how it's used. From within Omnicom (one of the large marketing & advertising agency groups) I can bring in a couple of my Annalect colleagues to talk about different types of measurement (including incrementality studies, digital-level attribution, and how they fit and are used with other techniques). This ticket is still open, so chairs please let me know if it's better to create a new one to schedule a session for that. Another thing I can offer is to get someone from one of our media agency teams who is hands on with DSPs and other buy-side tech; I suspect that might be most useful when we come back to talking about optimisation - again, we can pull that into another ticket, but it doesn't feel like that is a priority right now to schedule for a meeting. What might be helpful in the short term though is to start collecting questions that people in this group would want to ask of hands-on buy side folks. There are also some contacts I have at advertiser bodies who I'll introduce directly to the chairs to discuss participation, including providing another point of view on questions. (It would be unhelpful to hear only from an agency pov and not from advertisers themselves!) |
|
I spoke to a friend and former colleague, Laura, who used to pull levers on programmatic media buying. She's in the top 5 most admired "traders" I know of, though she's since moved on to solutions engineering (still in ads world). She said she'd be willing to join the next meeting and present / answer questions / participate in discussion. Are we still thinking we want to reserve time for this? |
|
@alextcone I would like to dig further into this with experts, but I know the time for this upcoming meeting will be unaccommodating. Can you plan to have them present in the meeting next month? We can open up a separate issue for that. @jaylett-annalect I would love to have those folks come in to our meeting next month. Let's plan on having a buy-side focused set of sessions with your contacts and @alextcone's. |
|
We'll plan on having the buy-side participants in a meeting closer to their timezone and we will open it up as a separate issue. |
|
An hour isn't anywhere near enough. Rather than boxing it in, we should
just let this go on as long as it's making progress.
…On Mon, May 16, 2022 at 2:42 PM Aram Zucker-Scharff < ***@***.***> wrote:
It sounds like there is enough here for more than an hour of conversation,
a lot of topics and back and forth in just this thread. I'm going to
reserve an hour on day one, but also place time in for open discussion on
measurement topics that this can overflow into on Day 2.
—
Reply to this email directly, view it on GitHub
<#51 (comment)>, or
unsubscribe
<https://github.com/notifications/unsubscribe-auth/AAIPLIMPG3SXO3YYNEXVONTVKK6NNANCNFSM5VOODGMQ>
.
You are receiving this because you were mentioned.Message ID:
***@***.***>
|
Agenda+: Measurement use-cases deep dive and requirements
It would be useful for the group to discuss the use-cases / requirements for attribution measurement. As a group, we dove into this topic straight into the deep end (crypto research). I propose we take a step back and make sure we're all on the same page with respect to what we want a measurement API to be able to accomplish.
Goal for this session would be some alignment on critical use cases, and where there isn't alignment, at least a mutual understanding of how various members prioritize certain use-cases. Ideally we as a group can align on solutions which aim to satisfy most of our highest priority use-case requirements.
Some use-cases that I think we should discuss:
The text was updated successfully, but these errors were encountered: