Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Feature: Add support for running migration in batches #7755

Open
wants to merge 39 commits into
base: epic/campaigns
Choose a base branch
from
Open
Show file tree
Hide file tree
Changes from 33 commits
Commits
Show all changes
39 commits
Select commit Hold shift + click to select a range
5bb5d6d
feature: BatchMigration interface
alaca Feb 21, 2025
6ee4163
refactor: make run method optional if BatchMigration interface is imp…
alaca Feb 21, 2025
783e27f
feature: add support for batch migrations
alaca Feb 21, 2025
da54e5f
feature: add enqueueAction method; refactor parameter position
alaca Feb 21, 2025
38d040e
feature: add new status - RUNNING
alaca Feb 21, 2025
9d498ab
refactor: migration run hook
alaca Feb 21, 2025
27d6d0d
refactor: use batch processing
alaca Feb 21, 2025
35a7290
feature: add isBatchMigration prop
alaca Feb 24, 2025
f79a040
feature: add endpoints for running batch migration and for reschedule…
alaca Feb 24, 2025
3311632
refactor: extend BatchMigration class
alaca Feb 24, 2025
cbdc32d
feature: add BaseMigration class
alaca Feb 24, 2025
761e21d
refactor: extend BaseMigration class
alaca Feb 24, 2025
9d07d6a
refactor: extend BaseMigration class
alaca Feb 24, 2025
20994e0
feature: add BatchMigrationRunner controller
alaca Feb 24, 2025
ea5662e
refactor: add check for BaseMigration
alaca Feb 24, 2025
2c3fa32
refactor: use BatchMigrationRunner for running batch migrations
alaca Feb 24, 2025
10afdac
refactor: run migrations on action_scheduler_init hook
alaca Feb 24, 2025
698f631
refactor: remove method and fix parameter position
alaca Feb 24, 2025
5ff0d21
feature: add support for running batch migrations and continue failed…
alaca Feb 24, 2025
a7c2c42
refactor: get failed actions by status
alaca Feb 24, 2025
990bbf9
feature: add incomplete status
alaca Feb 24, 2025
972e7fd
feature: add incomplete and running status
alaca Feb 24, 2025
b7d31e3
Merge branch 'refs/heads/epic/campaigns' into feature/batch-migrations
alaca Feb 24, 2025
b100127
chore: update docblock
alaca Feb 24, 2025
58396d9
feature: add items count check
alaca Feb 24, 2025
16b6af6
refactor: use time instead strtotime
alaca Feb 24, 2025
b10cfa3
fix: migration sorting
alaca Feb 24, 2025
50b8832
fix: migrations timestamps
alaca Feb 24, 2025
8a974b9
refactor: use time instead strtotime
alaca Feb 24, 2025
c620062
fix: migration sorting
alaca Feb 24, 2025
070b23e
Merge remote-tracking branch 'origin/feature/batch-migrations' into f…
alaca Feb 24, 2025
069d3e2
refactor: use BaseMigration class
alaca Feb 24, 2025
d5ddbc5
Merge branch 'refs/heads/epic/campaigns' into feature/batch-migrations
alaca Feb 24, 2025
dc5bf12
feature: add new methods
alaca Feb 25, 2025
6698d31
feature: last migration step check
alaca Feb 25, 2025
1e3634f
refactor: migration sort
alaca Feb 25, 2025
2721987
refactor: cleanup
alaca Feb 25, 2025
be3c08a
feature: implement new method for batch processing
alaca Feb 25, 2025
475d608
feature: add readme
alaca Feb 27, 2025
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
3 changes: 3 additions & 0 deletions src/API/Endpoints/Migrations/GetMigrations.php
Original file line number Diff line number Diff line change
Expand Up @@ -2,6 +2,7 @@

namespace Give\API\Endpoints\Migrations;

use Give\Framework\Migrations\Contracts\BatchMigration;
use Give\Framework\Migrations\Contracts\Migration;
use Give\Framework\Migrations\MigrationsRegister;
use Give\MigrationLog\Helpers\MigrationHelper;
Expand Down Expand Up @@ -161,6 +162,7 @@ public function handleRequest(WP_REST_Request $request)
'run_order' => $this->migrationHelper->getRunOrderForMigration($migration->getId()),
'source' => $migrationClass::source(),
'title' => $migrationClass::title(),
'isBatchMigration' => is_subclass_of($migrationClass, BatchMigration::class)
];
}

Expand All @@ -175,6 +177,7 @@ public function handleRequest(WP_REST_Request $request)
'run_order' => $this->migrationHelper->getRunOrderForMigration($migration::id()),
'source' => $migration::source(),
'title' => $migration::title(),
'isBatchMigration' => is_subclass_of($migration, BatchMigration::class)
];
}

Expand Down
136 changes: 119 additions & 17 deletions src/API/Endpoints/Migrations/RunMigration.php
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

A couple of notes/questions on this:

Namespace
The new RESTful routes in GiveWP are going to use the givewp/v3 namespace, and I believe exist in a V3 sub-domain — double-check with @jonwaldstein on this

Starting / Restarting Migrations
The concept of rescheduling failed actions is interesting. If I recall correctly, the system works such that only a single migration can actually be failed at a time, which then prevents subsequent migrations from running. So resuming migrations really means restarting the failed migration, and once it's successful then the subsequent migrations will follow. So they're not "failed", they're simply blocked.

This makes me wonder if we really need to introduce the concept of "failed" to the API, or if it's simply sufficient to say that a given migration can be "started". If it's an automatic migration then this is usually unnecessary, but can be triggered in the admin if necessary. For a batch migration, this may be a more necessary step — and it may not really matter whether the migration has yet to run or failed to run and needs to be attempted again.

RESTful API
This makes me think we could actually support a PUT/PATCH for the migration resource:

PATCH migrations/123
{
    "status": "run"
}

This makes it RESTful, and would even allow for JS Entities for interacting with the migration. Status would be a simple property of the resource, and the admin can change it to only this one status — wherein we also check that the existing status is in a state that's able to be switched to run (as opposed to try to run a running migration).

Copy link
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

If I recall correctly, the system works such that only a single migration can actually be failed at a time, which then prevents subsequent migrations from running

Are you talking about our migration system or AS?

This is true for our migration system, but not for AS.

The thing with Action Scheduler is that it doesn't care about failed/canceled actions. AS runs actions concurrently so there can be many failed/canceled actions at the same time, but that doesn't stop AS from running the rest. Once an action is failed or canceled, AS will not try to run it again. Actually, it will delete all of them after some time. Also, devs behind AS are recommending if the action fails, but you need that action to run, you should reschedule it.

I believe we can't really leave any orphaned data like in this example - Assign donations to a campaign. That's why I introduced this. Also, if batch migration has failed/canceled actions, it's not marked as failed, it is marked as incomplete.

Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I'm talking about the migration system. I guess we do need to think about what happens if batch migration enters into a partial state — wherein some of the steps have succeeded and others have failed. 🤔

To be clear, at no point should two migrations be running — even if they're batch migrations.

Copy link
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

To be clear, at no point should two migrations be running — even if they're batch migrations.

Yeah, that is not happening. Not even in our current broken system 😅

Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Happy to hop on a call, at a time that isn't interrupting work, and think through this a bit more.

Original file line number Diff line number Diff line change
Expand Up @@ -3,6 +3,10 @@
namespace Give\API\Endpoints\Migrations;

use Exception;
use Give\Framework\Database\DB;
use Give\Framework\Migrations\Contracts\BatchMigration;
use Give\Framework\Migrations\Contracts\Migration;
use Give\Framework\Migrations\Controllers\BatchMigrationRunner;
use Give\Framework\Migrations\MigrationsRegister;
use Give\MigrationLog\MigrationLogFactory;
use Give\MigrationLog\MigrationLogStatus;
Expand All @@ -11,16 +15,13 @@

/**
* Class RunMigration
* @package Give\API\Endpoints\Migrations
* @package Give\API\Endpoints\Migrations
*
* @since 2.10.0
* @unreleased run batch migrations
* @since 2.10.0
*/
class RunMigration extends Endpoint
{

/** @var string */
protected $endpoint = 'migrations/run-migration';

/**
* @var MigrationsRegister
*/
Expand All @@ -34,7 +35,7 @@ class RunMigration extends Endpoint
/**
* RunMigration constructor.
*
* @param MigrationsRegister $migrationsRegister
* @param MigrationsRegister $migrationsRegister
* @param MigrationLogFactory $migrationLogFactory
*/
public function __construct(
Expand All @@ -52,23 +53,58 @@ public function registerRoute()
{
register_rest_route(
'give-api/v2',
$this->endpoint,
'migrations/run-migration',
[
[
'methods' => 'POST',
'callback' => [$this, 'handleRequest'],
'callback' => [$this, 'runMigration'],
'permission_callback' => [$this, 'permissionsCheck'],
'args' => [
'id' => [
'validate_callback' => function ($param) {
return ! empty(trim($param));
},
'type' => 'string',
'required' => true,
],
],
],
'schema' => [$this, 'getSchema'],
]
);

register_rest_route(
'give-api/v2',
'migrations/run-batch-migration',
[
[
'methods' => 'POST',
'callback' => [$this, 'runBatchMigration'],
'permission_callback' => [$this, 'permissionsCheck'],
'args' => [
'id' => [
'type' => 'string',
'required' => true,
],
],
],
]
);

register_rest_route(
'give-api/v2',
'migrations/reschedule-failed-actions',
[
[
'methods' => 'POST',
'callback' => [$this, 'rescheduleFailedActions'],
'permission_callback' => [$this, 'permissionsCheck'],
'args' => [
'id' => [
'type' => 'string',
'required' => true,
],
],
],
]
);
}

/**
Expand All @@ -94,29 +130,31 @@ public function getSchema()
*
* @return WP_REST_Response
*/
public function handleRequest(WP_REST_Request $request)
public function runMigration(WP_REST_Request $request): WP_REST_Response
{
global $wpdb;
$migrationId = $request->get_param('id');
$migrationLog = $this->migrationLogFactory->make($migrationId);

// Begin transaction
$wpdb->query('START TRANSACTION');
DB::beginTransaction();

try {
$migrationClass = $this->migrationRegister->getMigration($migrationId);
/**
* @var Migration $migration
*/
$migration = give($migrationClass);
$migration->run();
// Save migration status
$migrationLog->setStatus(MigrationLogStatus::SUCCESS);
$migrationLog->setError(null);
$migrationLog->save();

$wpdb->query('COMMIT');
DB::commit();

return new WP_REST_Response(['status' => true]);
} catch (Exception $exception) {
$wpdb->query('ROLLBACK');
DB::rollback();

$migrationLog->setStatus(MigrationLogStatus::FAILED);
$migrationLog->setError($exception);
Expand All @@ -131,4 +169,68 @@ public function handleRequest(WP_REST_Request $request)
);
}


/**
* Run batch migration
*
* @unreleased
*/
public function runBatchMigration(WP_REST_Request $request): WP_REST_Response
{
$migrationId = $request->get_param('id');
$migrationClass = $this->migrationRegister->getMigration($migrationId);

if ( ! is_subclass_of($migrationClass, BatchMigration::class)) {
return new WP_REST_Response([
'status' => false,
'message' => 'Migration is not an instance of ' . BatchMigration::class,
]);
}

try {
// We are not running migration directly,
// we just have to set migration status to PENDING and Migration Runner will handle it
$migrationLog = $this->migrationLogFactory->make($migrationId);
$migrationLog->setStatus(MigrationLogStatus::PENDING);
$migrationLog->save();
} catch (Exception $e) {
return new WP_REST_Response([
'status' => false,
'message' => $e->getMessage(),
]);
}

return new WP_REST_Response(['status' => true]);
}

/**
* Reschedule failed actions
*
* @unreleased
*/
public function rescheduleFailedActions(WP_REST_Request $request): WP_REST_Response
{
$migrationId = $request->get_param('id');
$migrationClass = $this->migrationRegister->getMigration($migrationId);
$migration = give($migrationClass);

if ( ! is_subclass_of($migration, BatchMigration::class)) {
return new WP_REST_Response([
'status' => false,
'message' => 'Migration is not an instance of ' . BatchMigration::class,
]);
}

try {
(new BatchMigrationRunner($migration))->rescheduleFailedActions();
} catch (Exception $e) {
return new WP_REST_Response([
'status' => false,
'message' => $e->getMessage(),
]);
}

return new WP_REST_Response(['status' => true]);
}

}
31 changes: 26 additions & 5 deletions src/Campaigns/Migrations/Donations/AddCampaignId.php
Original file line number Diff line number Diff line change
Expand Up @@ -5,13 +5,13 @@
use Give\Donations\ValueObjects\DonationMetaKeys;
use Give\Framework\Database\DB;
use Give\Framework\Database\Exceptions\DatabaseQueryException;
use Give\Framework\Migrations\Contracts\Migration;
use Give\Framework\Migrations\Contracts\BatchMigration;
use Give\Framework\Migrations\Exceptions\DatabaseMigrationException;

/**
* @unreleased
*/
class AddCampaignId extends Migration
class AddCampaignId extends BatchMigration
{
/**
* @inheritDoc
Expand All @@ -34,14 +34,14 @@ public static function title(): string
*/
public static function timestamp(): string
{
return strtotime('2024-11-22 00:00:00');
return time();
}

/**
* @inheritDoc
* @throws DatabaseMigrationException
*/
public function run()
public function runBatch($batchNumber)
{
$relationships = [];

Expand All @@ -63,6 +63,8 @@ public function run()
[DonationMetaKeys::FORM_ID(), 'formId']
)
->where('post_type', 'give_payment')
->offset($batchNumber)
->limit($this->getBatchSize())
->getAll();

$donationMeta = [];
Expand All @@ -86,7 +88,26 @@ public function run()
->insert($donationMeta, ['%d', '%s', '%d']);
}
} catch (DatabaseQueryException $exception) {
throw new DatabaseMigrationException("An error occurred while adding campaign ID to the donation meta table", 0, $exception);
throw new DatabaseMigrationException("An error occurred while adding campaign ID to the donation meta table",
0, $exception);
}
}

/**
* @inheritDoc
*/
public function getItemsCount(): int
{
return DB::table('posts')
->where('post_type', 'give_payment')
->count();
}

/**
* @inheritDoc
*/
public function getBatchSize(): int
{
return 50;
}
}
2 changes: 1 addition & 1 deletion src/DonationForms/Migrations/CleanMultipleSlashesOnDB.php
Original file line number Diff line number Diff line change
Expand Up @@ -61,7 +61,7 @@ public static function title()
*/
public static function timestamp()
{
return strtotime('2023-20-11');
return strtotime('2023-11-20');
}

/**
Expand Down
Original file line number Diff line number Diff line change
Expand Up @@ -58,6 +58,6 @@ public static function title()
*/
public static function timestamp()
{
return strtotime('2022-19-12');
return strtotime('2022-12-19');
}
}
2 changes: 1 addition & 1 deletion src/Donations/Migrations/UnserializeTitlePrefix.php
Original file line number Diff line number Diff line change
Expand Up @@ -54,6 +54,6 @@ public static function title()
*/
public static function timestamp()
{
return strtotime('2024-23-10');
return strtotime('2024-10-23');
}
}
2 changes: 1 addition & 1 deletion src/Donors/Migrations/AddPhoneColumn.php
Original file line number Diff line number Diff line change
Expand Up @@ -55,6 +55,6 @@ public static function title(): string
*/
public static function timestamp()
{
return strtotime('2024-26-03');
return strtotime('2024-03-26');
}
}
61 changes: 61 additions & 0 deletions src/Framework/Migrations/Contracts/BaseMigration.php
Original file line number Diff line number Diff line change
@@ -0,0 +1,61 @@
<?php

namespace Give\Framework\Migrations\Contracts;

use Give\Framework\Exceptions\Primitives\RuntimeException;

/**
* Base Migration class
*
* @unreleased
*/
abstract class BaseMigration
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I think it could be useful to add another method or a class constant:

public static function startAutomatically(): bool {
    return true;
}

In the not too distant future we'll have migrations that we want to prompt the user to begin. Those migrations should set this to false. All it does is that if the MigrationsRunner encounters this as false, then it simply stops and doesn't proceed.

No need to build a UI or anything for this now, as it's outside of the scope of what you're doing. But it came to mind as something we'll want in this class.

Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Now that I say all that, maybe it's best to not introduce this now and instead introduce it when we build the UI, later... 🤔

Copy link
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

It's a simple change, but yeah, we can add that later.

{
/**
* Return a unique identifier for the migration
*
* @return string
*/
public static function id()
{
throw new RuntimeException('A unique ID must be provided for the migration');
}

/**
* Return a Unix Timestamp for when the migration was created
*
* Example: strtotime( '2020-09-16 12:30:00')
*
* @since 2.9.0
*
* @return int Unix timestamp for when the migration was created
*/
public static function timestamp()
{
throw new RuntimeException('This method must be overridden to return a valid unix timestamp');
}

/**
* Return migration title
*
* @since 2.10.0
*
* @return string
*/
public static function title()
{
return static::id();
}

/**
* Return migration source
*
* @since 2.10.0
*
* @return string
*/
public static function source()
{
return esc_html__('GiveWP Core', 'give');
}
}
Loading
Loading