Page MenuHomePhabricator

Modern Event Platform: Stream Intake Service: Migrate eventlogging-service-eventbus events to eventgate-main
Closed, ResolvedPublic21 Estimated Story Points

Description

This includes making the minor schema changes needed, i.e. meta.schema_uri -> $schema and meta.topic -> meta.stream.

Event streams to migrate in order of lowest to highest risk:

  • user-blocks-change
  • revision-tags-change
  • page-properties-change
  • page-links-change
  • page-restrictions-change
  • campaign-{create,change,delete}
  • page-move
  • page-delete/undelete
  • page-create
  • recentchange - public
  • revision-visibility-change - security implications
  • revision-create - this is basically the starter event that is beginning most of the update chains for change-prop
  • resource-change (Mediawiki EventBus)
  • resource-change (RESTBase)
  • revision-score (change-prop)

Before we can decom eventlogging-service-eventbus, we also need to complete:

Details

SubjectRepoBranchLines +/-
operations/puppetproduction+0 -22
operations/puppetproduction+6 -5
operations/mediawiki-configmaster+3 -0
operations/puppetproduction+37 -98
operations/puppetproduction+20 -28
operations/deployment-chartsmaster+1 -1
operations/deployment-chartsmaster+9 -0
mediawiki/services/change-propagation/deploymaster+2 -2
operations/puppetproduction+1 -1
mediawiki/extensions/EventBusmaster+1 -1
operations/mediawiki-configmaster+3 -0
operations/puppetproduction+1 -1
operations/mediawiki-configmaster+9 -0
operations/mediawiki-configmaster+3 -0
operations/mediawiki-configmaster+3 -0
operations/mediawiki-configmaster+8 -5
operations/puppetproduction+5 -4
operations/mediawiki-configmaster+12 -0
operations/mediawiki-configmaster+3 -0
mediawiki/vagrantmaster+76 -44
operations/puppetproduction+15 -2
operations/mediawiki-configmaster+3 -0
operations/mediawiki-configmaster+3 -0
operations/mediawiki-configmaster+3 -0
mediawiki/event-schemasmaster+5 -5
operations/mediawiki-configmaster+3 -0
mediawiki/event-schemasmaster+1 -1
mediawiki/event-schemasmaster+2 -3
mediawiki/extensions/EventBusmaster+73 -48
operations/mediawiki-configmaster+0 -3
operations/mediawiki-configmaster+1 -1
operations/mediawiki-configmaster+10 -0
mediawiki/event-schemasmaster+416 -2
mediawiki/event-schemasmaster+2 K -94
Show related patches Customize query in gerrit

Related Objects

StatusSubtypeAssignedTask
ResolvedOttomata
ResolvedOttomata
ResolvedOttomata
Resolved Pchelolo
ResolvedOttomata
Resolvedakosiaris
ResolvedOttomata
ResolvedOttomata
Resolved Pchelolo
ResolvedOttomata
Duplicate Pchelolo
Resolved Pchelolo
ResolvedHalfak
Resolved Pchelolo
Resolved Pchelolo
Resolved Pchelolo
ResolvedOttomata
ResolvedOttomata

Event Timeline

There are a very large number of changes, so older changes are hidden. Show Older Changes

Change 521328 had a related patch set uploaded (by Ottomata; owner: Ottomata):
[operations/puppet@production] Refine mediawiki_page* with schema aware Refine job

https://gerrit.wikimedia.org/r/521328

Change 521328 merged by Ottomata:
[operations/puppet@production] Refine mediawiki_page* with schema aware Refine job

https://gerrit.wikimedia.org/r/521328

Change 521517 had a related patch set uploaded (by Ottomata; owner: Ottomata):
[operations/mediawiki-config@master] Migrate mediawiki.recentchange stream to eventgate-main

https://gerrit.wikimedia.org/r/521517

Change 521528 had a related patch set uploaded (by Ottomata; owner: Ottomata):
[operations/mediawiki-config@master] Move wgRCFeeds settings from CommonSettings to InitialiseSettings

https://gerrit.wikimedia.org/r/521528

Change 521528 merged by jenkins-bot:
[operations/mediawiki-config@master] Use wgEventServiceStreamConfig to configure wgRCFeeds['eventbus']

https://gerrit.wikimedia.org/r/521528

Mentioned in SAL (#wikimedia-operations) [2019-07-10T19:26:28Z] <otto@deploy1001> Synchronized wmf-config/CommonSettings.php: Use wgEventServiceStreamConfig to configure wgRCFeeds eventbus. No-op in prod. - T211248 (duration: 00m 58s)

Change 521517 merged by jenkins-bot:
[operations/mediawiki-config@master] Migrate mediawiki.recentchange stream to eventgate-main

https://gerrit.wikimedia.org/r/521517

Mentioned in SAL (#wikimedia-operations) [2019-07-10T19:32:22Z] <otto@deploy1001> Synchronized wmf-config/InitialiseSettings.php: Produce recentchange stream to eventgate-main - T211248 (duration: 00m 57s)

@DStrine, I need some help from someone who knows how to make changes to centralnotice campaigns to test that these events work after migration. Who should I ask?

@DStrine, I need some help from someone who knows how to make changes to centralnotice campaigns to test that these events work after migration. Who should I ask?

Hi! I can help with this, sorry for the delay in replying via other channels...! Thanks!!

Change 523180 had a related patch set uploaded (by Ottomata; owner: Ottomata):
[operations/mediawiki-config@master] Produce revision-visibility-change event to eventgate-main

https://gerrit.wikimedia.org/r/523180

Change 523180 merged by jenkins-bot:
[operations/mediawiki-config@master] Produce revision-visibility-change event to eventgate-main

https://gerrit.wikimedia.org/r/523180

Mentioned in SAL (#wikimedia-operations) [2019-07-15T16:57:13Z] <otto@deploy1001> Synchronized wmf-config/InitialiseSettings.php: Produce revision-visibility-change stream to eventgate-main - T211248 (duration: 00m 49s)

Change 523232 had a related patch set uploaded (by Ottomata; owner: Ottomata):
[operations/mediawiki-config@master] Produce revision-create stream to eventgate-main

https://gerrit.wikimedia.org/r/523232

Change 520019 merged by Ottomata:
[operations/mediawiki-config@master] Produce centralnotice.campaign-* streams to eventgate-main

https://gerrit.wikimedia.org/r/520019

Mentioned in SAL (#wikimedia-operations) [2019-07-16T17:39:40Z] <otto@deploy1001> Synchronized wmf-config/InitialiseSettings.php: Produce centralnotice.campaign-* streams to eventgate-main - T211248 (duration: 00m 55s)

Change 523787 had a related patch set uploaded (by Ottomata; owner: Ottomata):
[operations/puppet@production] Move revision_visibility_change Refine job to schema aware version

https://gerrit.wikimedia.org/r/523787

Change 523787 merged by Ottomata:
[operations/puppet@production] Move revision_visibility_change Refine job to schema aware version

https://gerrit.wikimedia.org/r/523787

Change 523232 merged by Ottomata:
[operations/mediawiki-config@master] Produce revision-create stream to eventgate-main

https://gerrit.wikimedia.org/r/523232

Mentioned in SAL (#wikimedia-operations) [2019-07-16T18:22:59Z] <otto@deploy1001> Synchronized wmf-config/InitialiseSettings.php: Produce revision-create stream to eventgate-main - T211248 (duration: 00m 54s)

Mentioned in SAL (#wikimedia-operations) [2019-07-16T18:46:59Z] <otto@deploy1001> Synchronized wmf-config/InitialiseSettings.php: retry - Produce revision-create stream to eventgate-main - T211248 (duration: 00m 54s)

Change 523791 had a related patch set uploaded (by Ottomata; owner: Ottomata):
[operations/puppet@production] Refine mediawiki_revision_create events using schema aware Refine job

https://gerrit.wikimedia.org/r/523791

Change 523794 had a related patch set uploaded (by Ottomata; owner: Ottomata):
[mediawiki/extensions/EventBus@master] Use proper schema url for resource_change event

https://gerrit.wikimedia.org/r/523794

Change 523794 merged by Ottomata:
[mediawiki/extensions/EventBus@master] Use proper schema url for resource_change event

https://gerrit.wikimedia.org/r/523794

Ottomata renamed this task from Modern Event Platform: Stream Intake Service: Migrate Mediawiki Eventbus events to eventgate-main to Modern Event Platform: Stream Intake Service: Migrate eventlogging-service-eventbus events to eventgate-main.Jul 17 2019, 8:28 PM
Ottomata updated the task description. (Show Details)

Change 523791 merged by Ottomata:
[operations/puppet@production] Refine mediawiki_revision_create events using schema aware Refine job

https://gerrit.wikimedia.org/r/523791

Change 524055 had a related patch set uploaded (by Ppchelko; owner: Ppchelko):
[mediawiki/services/change-propagation/deploy@master] Switch revision-create to eventgate.

https://gerrit.wikimedia.org/r/524055

Change 524055 had a related patch set uploaded (by Ppchelko; owner: Ppchelko):
[mediawiki/services/change-propagation/deploy@master] Switch revision-score to eventgate.

https://gerrit.wikimedia.org/r/524055

Change 524055 merged by Ppchelko:
[mediawiki/services/change-propagation/deploy@master] Switch revision-score to eventgate.

https://gerrit.wikimedia.org/r/524055

Mentioned in SAL (#wikimedia-operations) [2019-07-22T19:07:30Z] <ppchelko@deploy1001> Started deploy [changeprop/deploy@3f8aad2]: Switch revision-score to eventgate T211248

Mentioned in SAL (#wikimedia-operations) [2019-07-22T19:09:01Z] <ppchelko@deploy1001> Finished deploy [changeprop/deploy@3f8aad2]: Switch revision-score to eventgate T211248 (duration: 01m 31s)

Mentioned in SAL (#wikimedia-operations) [2019-07-22T21:04:57Z] <ppchelko@deploy1001> Started deploy [restbase/deploy@ea10fa5]: Switch event production to eventgate T211248

Change 524922 had a related patch set uploaded (by Ottomata; owner: Ottomata):
[operations/deployment-charts@master] eventgate-main configure change-prop.transcludes.resource-change stream

https://gerrit.wikimedia.org/r/524922

Change 524922 merged by Ottomata:
[operations/deployment-charts@master] eventgate-main configure change-prop.transcludes.resource-change stream

https://gerrit.wikimedia.org/r/524922

Mentioned in SAL (#wikimedia-operations) [2019-07-22T21:21:11Z] <ppchelko@deploy1001> Finished deploy [restbase/deploy@ea10fa5]: Switch event production to eventgate T211248 (duration: 16m 14s)

Mentioned in SAL (#wikimedia-operations) [2019-07-22T21:25:40Z] <ppchelko@deploy1001> Started deploy [restbase/deploy@9a99b17]: Rollback: Switch event production to eventgate T211248

Mentioned in SAL (#wikimedia-operations) [2019-07-22T21:38:39Z] <ppchelko@deploy1001> Finished deploy [restbase/deploy@9a99b17]: Rollback: Switch event production to eventgate T211248 (duration: 13m 01s)

Mentioned in SAL (#wikimedia-operations) [2019-07-23T14:39:34Z] <ppchelko@deploy1001> Started deploy [restbase/deploy@ea10fa5]: Switch event production to eventgate T211248, attempt 2

Mentioned in SAL (#wikimedia-operations) [2019-07-23T14:52:42Z] <ppchelko@deploy1001> Finished deploy [restbase/deploy@ea10fa5]: Switch event production to eventgate T211248, attempt 2 (duration: 13m 08s)

Change 525110 had a related patch set uploaded (by Ottomata; owner: Ottomata):
[operations/deployment-charts@master] Use proper main-codfw Kafka cluster for eventgate-main in codfw

https://gerrit.wikimedia.org/r/525110

Change 525110 merged by Ottomata:
[operations/deployment-charts@master] Use proper main-codfw Kafka cluster for eventgate-main in codfw

https://gerrit.wikimedia.org/r/525110

Change 525114 had a related patch set uploaded (by Ppchelko; owner: Ppchelko):
[operations/puppet@production] Clean up eventlogging_service_uri from RESTBase profile.

https://gerrit.wikimedia.org/r/525114

Change 525121 had a related patch set uploaded (by Ppchelko; owner: Ppchelko):
[operations/puppet@production] Clean up eventlogging_service_uri from maps.

https://gerrit.wikimedia.org/r/525121

Change 525121 abandoned by Ppchelko:
Clean up eventlogging_service_uri from maps.

Reason:
Superseded by I06a90a199c5832a7fe2505bffa5c5cf40281ff98

https://gerrit.wikimedia.org/r/525121

Change 525121 restored by Ppchelko:
Clean up eventlogging_service_uri from maps.

https://gerrit.wikimedia.org/r/525121

Change 525114 merged by Ottomata:
[operations/puppet@production] Clean up eventlogging_service_uri from RESTBase profile.

https://gerrit.wikimedia.org/r/525114

Change 525121 merged by Ottomata:
[operations/puppet@production] Clean up eventlogging_service_uri from maps.

https://gerrit.wikimedia.org/r/525121

Change 525854 had a related patch set uploaded (by Ppchelko; owner: Ppchelko):
[operations/mediawiki-config@master] [EventBus] Switch resource_change event to eventgate.

https://gerrit.wikimedia.org/r/525854

Change 525854 merged by Ottomata:
[operations/mediawiki-config@master] [EventBus] Switch resource_change event to eventgate.

https://gerrit.wikimedia.org/r/525854

Mentioned in SAL (#wikimedia-operations) [2019-07-29T15:30:46Z] <otto@deploy1001> Synchronized wmf-config/InitialiseSettings.php: Produce resource_change stream to eventgate-main - T211248 (duration: 00m 47s)

Mentioned in SAL (#wikimedia-operations) [2019-07-29T15:35:26Z] <otto@deploy1001> Synchronized wmf-config/InitialiseSettings.php: Retry - Produce resource_change stream to eventgate-main - T211248 (duration: 00m 46s)

Change 526180 had a related patch set uploaded (by Ottomata; owner: Ottomata):
[operations/puppet@production] Use schema aware refine for revision score and resource change

https://gerrit.wikimedia.org/r/526180

Change 526180 merged by Ottomata:
[operations/puppet@production] Use schema aware refine for revision score and resource change

https://gerrit.wikimedia.org/r/526180

Change 526435 had a related patch set uploaded (by Ottomata; owner: Ottomata):
[operations/puppet@production] Remove unsed refine job refine_eventlogging_eventbus

https://gerrit.wikimedia.org/r/526435

Change 526435 merged by Ottomata:
[operations/puppet@production] Remove unsed refine job refine_eventlogging_eventbus

https://gerrit.wikimedia.org/r/526435

Change 534637 had a related patch set uploaded (by Ppchelko; owner: Ppchelko):
[operations/mediawiki-config@master] Remove references to eventlogging-service.

https://gerrit.wikimedia.org/r/534637

Ottomata set the point value for this task to 13.

Let's do the wave!!!!!