Agaric Collective: Workflows and benefits of managing Drupal migrations as configuration entities

Planet Drupal - Sat, 2019/08/24 - 5:00am

In the last blog post we were introduced to managing migration as configuration entities using Migrate Plus. Today, we will present some benefits and potential drawbacks of this approach. We will also show a recommended workflow for working with migration as configuration. Let’s get started.

What is the benefit of managing migration as configurations?

At first sight, there does not seem to be a big difference between defining migrations as code or configuration. You can certainly do a lot without using Migrate Plus’ configuration entities. The series so far contains many examples of managing migrations as code. So, what are the benefits of adopting s configuration entities?

The configuration management system is one of the major features that was introduced in Drupal 8. It provides the ability to export all your site’s configuration to files. These files can be added to version control and deployed to different environments. The system has evolved a lot in the last few years, and many workflows and best practices have been established to manage configuration. On top of Drupal core’s incremental improvements, a big ecosystem has sprung in terms of contributed modules. When you manage migrations via configuration, you can leverage those tools and workflows.

Here are a few use cases of what is possible:

  • When migrations are managed in code, you need file system access to make any changes. Using configuration entities allows site administrators to customize or change the migration via the user interface. This is not about rewriting all the migrations. That should happen during development and never on production environments. But it is possible to tweak certain options. For example, administrators could change the location to the file that is going to be migrated, be it a local file or on a remote server.
  • When writing migrations, it is very likely that you will work on a subset of the data that will eventually be used to get content into the production environment.  Having migrations as configuration allow you to override part of the migration definition per environment. You could use the Configuration Split module to configure different source files or paths per environment. For example, you could link to a small sample of the data in development, a larger sample in staging, and the complete dataset in production.
  • It would be possible to provide extra configuration options via the user interface. In the article about adding HTTP authentication to fetch remote JSON and XML files, the credentials were hardcoded in the migration definition file. That is less than ideal and exposes sensitive information. An alternative would be to provide a configuration form in the administration interface for the credentials to be added. Then, the submitted values could be injected into the configuration for the migration. Again, you could make use of contrib modules like Configuration Split to make sure those credentials are never exported with the rest of your site’s configuration.
  • You could provide a user interface to upload migration source files. In fact, the Migrate source UI module does exactly this. It exposes an administration interface where you have a file field to upload a CSV file. In the same interface, you get a list of supported migrations in the system. This allows a site administrator to manually upload a file to run the migration against. Note: The module is supposed to work with JSON and XML migrations. It did not work during my tests. I opened this issue to follow up on this.

These are some examples, but many more possibilities are available. The point is that you have the whole configuration management ecosystem at your disposal. Do you have another example? Please share it in the comments.

Are there any drawbacks?

Managing configuration as configuration adds an extra layer of abstraction in the migration process. This adds a bit of complexity. For example:

  • Now you have to keep the uuid and id keys in sync. This might not seem like a big issue, but it is something to pay attention to.
  • When you work with migrations groups (explained in the next article), your migration definition could live in more file.
  • The configuration management system has its own restrictions and workflows that you need to follow, particularly for updates.
  • You need to be extra careful with your YAML syntax, especially if syncing configuration via the user interface. It is possible to import invalid configuration without getting an error. It is until the migration fails that you realize something is wrong.

Using configuration entities to define migrations certainly offers lots of benefits. But it requires being extra careful managing them.

Workflow for managing migrations as configuration entities

The configuration synchronization system has specific workflows to make changes to configuration entities. This imposes some restrictions in the way you make updates to the migration definitions. Explaining how to manage configuration could use another 31 days blog post series. ;-) For now, only a general overview will be presented. The general approach is similar to managing configuration as code. The main difference is what needs to be done for changes to the migration files to take effect.

You could use the “Configuration synchronization” administration interface at /admin/config/development/configuration. In it you have the option to export  or import a “full archive” containing all your site’s settings or a “single item” like a specific migration. This is one way to manage migrations as configuration entities which lets you find their UUIDs if not set initially. This approach can be followed by site administrators without requiring file system access. Nevertheless, it is less than ideal and error-prone. This is not the recommended way to manage migration configuration entities.

Another option is to use Drush or Drupal Console to synchronize your site’s configuration via the command line. Similarly to the user interface approach, you can export and import your full site configuration or only single elements. The recommendation is to do partial configuration imports so that only the migrations you are actively working on are updated.

Ideally, your site’s architecture is completed before the migration starts. In practice, you often work on the migration while other parts of the sites are being built. If you were to export and import the entire site’s configuration as you work on the migrations, you might inadvertently override unrelated pieces of configurations. For instance, this can lead to missing content types, changed field settings, and lots of frustration. That is why doing partial or single configuration imports is recommended. The following code snippet shows a basic Drupal workflow for managing migrations as configuration:

# 1) Run the migration. $ drush migrate:import udm_config_json_source_node_local # 2) Rollback migration because the expected results were not obtained. $ drush migrate:rollback udm_config_json_source_node_local # 3) Change the migration definition file in the "config/install" directory. # 4a) Sync configuration by folder using Drush. $ drush config:import --partial --source="modules/custom/ud_migrations/ud_migrations_config_json_source/config/install" # 4b) Sync configuration by file using Drupal Console. $ drupal config:import:single --file="modules/custom/ud_migrations/ud_migrations_config_json_source/config/install/migrate_plus.migration.udm_config_json_source_node_local.yml" # 5) Run the migration again. $ drush migrate:import udm_config_json_source_node_local

Note the use of the --partial and --source flags in the migration import command. Also, note that the path is relative to the current working directory from where the command is being issued. In this snippet, the value of the source flag is the directory holding your migrations. Be mindful if there are other non-migration related configurations in the same folder. If you need to be more granular, Drupal Console offers a command to import individual configuration files as shown in the previous snippet.

Note: Uninstalling and installing the module again will also apply any changes to your configuration. This might produce errors if the migration configuration entities are not removed automatically when the module is uninstalled. Read this article for details on how to do that.

What did you learn in today’s blog post? Did you know the know the benefits and trade-offs of managing migrations as configuration? Did you know what to do for changes in migration configuration entities to take effect? Share your answers in the comments. Also, I would be grateful if you share this blog post with others.

This blog post series, cross-posted at UnderstandDrupal.com as well as here on Agaric.coop, is made possible thanks to these generous sponsors. Contact Understand Drupal if your organization would like to support this documentation project, whether it is the migration series or other topics.

Read more and discuss at agaric.coop.

Categories:

Agaric Collective: Defining Drupal migrations as configuration entities with the Migrate Plus module

Planet Drupal - Sat, 2019/08/24 - 12:28am

Today, we are going to talk about how to manage migrations as configuration entities. This functionality is provided by the Migrate Plus module. First, we will explain the difference between managing migrations as code or configuration. Then, we will show how to convert existing migrations. Finally, we will talk about some important options to include in migration configuration entities. Let’s get started.

Drupal migrations: code or configuration?

So far, we have been managing migrations as code. This is functionality provided out of the box. You write the migration definition file in YAML format. Then, you place it in the migrations directory of your module. If you need to update the migration, you make the modifications to the files and then rebuild caches. More details on the workflow for migrations managed in code can be found in this article.

Migrate Plus offers an alternative to this approach. It allows you to manage migrations as configuration entities. You still use YAML files to write the migration definition files, but their location and workflow is different. They need to be placed in a config/install directory. If you need to update the migration,  you make the modifications to the files and then sync the configuration again. More details on this workflow can be found in this article.

There is one thing worth emphasizing. When managing migrations as code you need access to the file system to update and deploy the changes to the file. This is usually done by developers.  When managing migrations as configuration, you can make updates via the user interface as long as you have permissions to sync the site’s configuration. This is usually done by site administrators. You might still have to modify files depending on how you manage your configuration. But the point is that file system access to update migrations is optional. Although not recommended, you can write, modify, and execute the migrations entirely via the user interface.

Transitioning to configuration entities

To demonstrate how to transition from code to configuration entities, we are going to convert the JSON migration example. You can get the full code example at https://github.com/dinarcon/ud_migrations The module to enable is UD config JSON source migration whose machine name is udm_config_json_source. It comes with four migrations: udm_config_json_source_paragraph, udm_config_json_source_image, udm_config_json_source_node_local, and udm_config_json_source_node_remote.

The transition to configuration entities is a two step process. First, move the migration definition files from the migrations folder to a config/install folder. Second, rename the files so that they follow this pattern: migrate_plus.migration.[migration_id].yml. For example: migrate_plus.migration.udm_config_json_source_node_local.yml. And that’s it! Files placed in that directory following that pattern will be synced into Drupal’s active configuration when the module is installed for the first time (only). Note that changes to the files require a new synchronization operation for changes to take effect. Changing the files and rebuilding caches does not update the configuration as it was the case with migrations managed in code.

If you have the Migrate Plus module enabled, it will detect the migrations and you will be able to execute them. You can continue using the Drush commands provided the Migrate Run module. Alternatively, you can install the Migrate Tools module which provides Drush commands for running both types of migrations: code and configuration. Migrate Tools also offers a user interface for executing migrations. This user interface is only for migrations defined as configuration though. It is available at /admin/structure/migrate. For now, you can run the migrations using the following Drush command: drush migrate:import udm_config_json_source_node_local --execute-dependencies.

Note: For executing migrations in the command line, choose between Migrate Run or Migrate Tools. You pick one or the other, but not both as the commands provided by the two modules have the same name. Another thing to note is that the example uses Drush 9. There were major refactorings between versions 8 and 9 which included changes to the name of the commands.

UUIDs for migration configuration entities

When managing migrations as configuration, you can set extra options. Some are exposed by Migrate Plus while others come from Drupal’s configuration management system. Let’s see some examples.

The most important new option is defining a UUID for the migration definition file. This is optional, but adding one will greatly simplify the workflow to update migrations. The UUID is used to keep track of every piece of configuration in the system. When you add new configuration, Drupal will read the UUID value if provided and update that particular piece of configuration. Otherwise, it will create a UUID on the fly, attach it to the configuration definition, and then import it. That is why you want to set a UUID value manually. If changes need to be made, you want to update the same configuration, not create a new one. If no UUID was originally set, you can get the automatically created value by exporting the migration definition. The workflow for this is a bit complicated and error prone so always include a UUID with your migrations. This following snippet shows an example UUID:

uuid: b744190e-3a48-45c7-97a4-093099ba0547 id: udm_config_json_source_node_local label: 'UD migrations configuration example'

The UUID a string of 32 hexadecimal digits displayed in 5 groups. Each is separated by hyphens following this pattern: 8-4-4-4-12. In Drupal, two or more pieces of configuration cannot share the same value. Drupal will check the UUID and the type of configuration in sync operations. In this case the type is signaled by the migrate_plus.migration. prefix in the name of the migration definition file.

When using configuration entities, a single migration is identified by two different options. The uuid is used by the Drupal’s configuration system and the id is used by the Migrate API. Always make sure that this combination is kept the same when updating the files and syncing the configuration. Otherwise you might get hard to debug errors. Also, make sure you are importing the proper configuration type. The latter should not be something to worry about unless you utilize the user interface to export or import single configuration items.

Tip: If you do not have a UUID in advance for your migration, you can copy one from another piece of configuration and change some of the hexadecimal digits. Keep in mind that this could lead to a duplicated UUID if you happen to make the exact changes to the UUID in two separate files. Another option is to use a tool to generate the values. Searching online for UUID generators will yield many  tools for this.

Automatically deleting migration configuration entities

By default, configuration remains in the system even if the module that added it gets uninstalled. This can cause problems if your migration depends on custom migration plugins provided by your module. It is possible to enforce that migration entities get removed when your custom module is uninstalled. To do this, you leverage the dependencies option provided by Drupal’s configuration management system. The following snippet shows how to do it:

uuid: b744190e-3a48-45c7-97a4-093099ba0547 id: udm_config_json_source_node_local label: 'UD migrations configuration example' dependencies: enforced: module: - ud_migrations_config_json_source

You add the machine name of your module to dependencies > enforced > module array. This adds an enforced dependency on your own module. The effect is that the migration will be removed from Drupal’s active configuration when your custom module is uninstalled. Note that the top level dependencies array can have others keys in addition to enforced. For example: config and module. Learning more about them is left as an exercise for the curious reader.

It is important not to confuse the dependencies and migration_dependencies options. The former is provided by Drupal’s configuration management system and was just explained. The latter is provided by the Migrate API and is used to declare migrations that need be imported in advance. Read this article to know more about this feature. The following snippet shows an example:

uuid: b744190e-3a48-45c7-97a4-093099ba0547 id: udm_config_json_source_node_local label: 'UD migrations configuration example' dependencies: enforced: module: - ud_migrations_config_json_source migration_dependencies: required: - udm_config_json_source_image - udm_config_json_source_paragraph optional: []

What did you learn in today’s blog post? Did you know that you can manage migrations in two ways: code or configuration? Did you know that file name and location as well as workflows need to be adjusted depending on which approach you follow? Share your answers in the comments. Also, I would be grateful if you shared this blog post with others.

Next: Workflows and benefits of managing Drupal migrations as configuration entities

This blog post series, cross-posted at UnderstandDrupal.com as well as here on Agaric.coop, is made possible thanks to these generous sponsors. Contact Understand Drupal if your organization would like to support this documentation project, whether it is the migration series or other topics.

Read more and discuss at agaric.coop.

Categories:

Srijan Technologies: Drupal is Here to Increase Your Marketing Agility

Planet Drupal - Fri, 2019/08/23 - 2:40pm



Marketing department these days are feeling the heat to make processes faster, agile, and efficient in this fast-paced digital world. That’s where the concept of Drupal comes into the picture! 

Categories:

Srijan Technologies: OpenSocial Bounty Winner from Srijan is All Set for DrupalCon Amsterdam 😃

Planet Drupal - Fri, 2019/08/23 - 1:30pm



In June, OpenSocial launched one of its kind contest inviting developers from Drupal community to contribute and win a free ticket and flight to DrupalCon Amsterdam. Jyoti Singh, Sr. Drupal Developer, won the bounty and is all ready to visit the city of canals. ✨

A regular contributor at Srijan local code sprints, elated Jyoti says, “I am very excited right now but honestly wasn’t expecting anything when I participated. I did this because I love contributing and what better opportunity than getting DrupalCon sponsored in return”.

DrupalCon is an international conference where the Drupal community joins together for a full week dedicated to sharing and growing Drupal skills. DrupalCon Amsterdam happens from 28 - 31 October 2019.

What was the Contest? The contest required participants to:
  • Visit the bounty issue queue
  • For each issue credit on an issue marked with the 'OSBounty- DrupalConEU2019' label, participants would receive an entry ticket for the lottery. Only Open Social maintainers can add this label.
  • Once done, participants would be automatically added to the lottery for tickets to DrupalCon Amsterdam 2019.
Excited for Contributions at DrupalCon

A regular at DrupalCons, Jyoti looks forward to the code sprints and contribution days “I remember in Vienna (2017), there were areas dedicated to distributions and issues. This is a really good opportunity to meet the core distribution and module maintainers and many such important people in the community. Meeting them is inspiring, where you know about the road map of such distributions and plan ahead what you can deliver to your client. Afterparty is another place to enjoy and network.”

Categories:

clemens-tolboom commented on issue hechoendrupal/drupal-console#3939

On github - Fri, 2019/08/23 - 1:21pm
clemens-tolboom commented on issue hechoendrupal/drupal-console#3939 Aug 23, 2019 clemens-tolboom commented Aug 23, 2019

I was wondering too. I can run core /usr/local/bin/php vendor/phpunit/phpunit/phpunit \ --configuration core \ --configuration core/phpunit.xml.dis…

Srijan Technologies: Integrating Drupal with AWS Machine Learning

Planet Drupal - Fri, 2019/08/23 - 10:00am

With enterprises looking for ways to stay ahead of the curve in the growing digital age, machine learning is providing them with the needed boost for seamless digital customer experience.

Categories:

Agiledrop.com Blog: Interview with pinball wizard Greg Dunlap, Senior Digital Strategist at Lullabot

Planet Drupal - Fri, 2019/08/23 - 8:19am

This time we talked with Greg Dunlap, pinball wizard and Lullabot's senior digital strategist. We spoke of how satisfying it is to work on interesting things with the right people, the importance of the Drupal Diversity and Inclusion initiative, and the close similarities between the Drupal community and Greg's local pinball community.

READ MORE
Categories:

clemens-tolboom commented on pull request hechoendrupal/drupal-console#4139

On github - Fri, 2019/08/23 - 7:40am
clemens-tolboom commented on pull request hechoendrupal/drupal-console#4139 Aug 23, 2019 clemens-tolboom commented Aug 23, 2019

Are there tests available (for checking generated code)? ... I'm just busy doning #4138 :-)

Freelock : Better Care Network

Planet Drupal - Thu, 2019/08/22 - 11:15pm
Better Care Network Don Dill Thu, 08/22/2019 - 14:15

More to come soon....

Categories:

Dropsolid: Contributing to Open Source, what's your math?

Planet Drupal - Thu, 2019/08/22 - 8:58pm
22 Aug

1xINTERNET made a great post about calculating how much you are giving back and contributing to Open Source. Open Source makes our business model possible and for some people in the company, and certainly yours truly, made a career out of it. Our CEO even started Dropsolid as he thought we could approach this differently. If this makes a difference for us, why not for some of our employees as well, or could it even be a big impact in recruitment?

Delivering Open Digital Experiences with Open Source components on our own Dropsolid Platform is also is very interesting for clients. We are able to deliver quality projects for a budget someone once could only dream of. The downside is that you are not alone that has this magical ability, so it is a competitive landscape. The question then is how you can stand out, how can you make that difference.

Let's honor 1xINTERNET by doing our math.

How do we contribute

Our contribution to the Drupal project can be divided in the same 3 areas:

  • Community work
  • Event, Sponsorships, and memberships
  • Source code
Community work

During the last year and even this year we spent a lot of time helping the Belgian Drupal Community. We organised Drupal User Group meetings, helped with organising Drupalcamp Belgium and are currently actively involved in organising Drupal Developer Days. Next to that is yours truly also active with the European Splash Awards as an organisational member and active as Chair of the Belgian Drupal Community. We also had several speakers from Dropsolid on all these events and I've also volunteered as Drupalcon Track Chair or Co-Chair for the DevOps track, both Drupal Europe and Drupalcon Amsterdam. Attending those meetings during the day or evening takes time, but is well worth it. And next to that we gave back to the community by going to Drupalcamp Kiev and Drupal Mountain Camp to share the knowledge.

When taking all of the above, time spent on community activities adds up to around 1 FTE. This includes time spent to organise it, give back with speaker preparations and time spent to organise those sponsorships and facilitate our employees to attend. For your information, in 2018 we had on average 65 people working for Dropsolid.

Sponsorships and memberships

Since 2018 we invested in the following events and memberships

  • Silver Sponsor for Drupal Europe 2018
  • Gold Sponsor for Drupalcamp Belgium
  • Gold Drupaljam XL 2019
  • Diamond Sponsor Drupalcon Amsterdam 2019
  • Organisation Member of Belgian Drupal Community
  • Organisation Member of the Drupal Association
  • Supporting Partner of the Drupal Association
  • Donation to the Promote Drupal Fund
  • Employ a Provisional member of the Drupal Security Team

in total we spent close to 1% of our total yearly spent in sponsorships, travel to and memberships related to the Drupal project.

Drupal.org contributions

Without a doubt we actively contribute back to the Drupal.org source code. Some contributions are also meetings from the community events and other important milestones that matter for the community. All efforts matter, not just code.

We contributed our complete install profile Dropsolid Rocketship and all our paragraphs that come with it. Dropsolid has a dedicated R&D team that consists out of 8 people. This team has a Drupal Backend & Frontend R&D Engineer, a couple DevOps engineers, 2 machine learning geniuses and myself. Next to that we supported one of our employees in his personal pet project drupalcontributions.org. Let's take a look at that site to come up with some numbers.

We support over 30 projects, have over 207 credits, with just 58 credits in the last 3 months and 181 comments in the last 3 months. I dare to say that we are ramping it up.

We currently rank nr. 42 of the organisations that contribute most back to Drupal in the last 3 months. Compared to 1xINTERNET we have a bigger headcount, but nevertheless I am equally proud, just as they are, to see this happen and to make an impact. 

Not only do we contribute with our own agenda, we also contract Swentel to spend just 1 day a month on Drupal contributions without any agreed agenda. By just supporting 1 day a month, this accounted to 17 comments in the last 30 days alone. To make a point, just counting credits isn't fair. It's how you come together and find a common solution. Communication is key and we support this with hard earned money.

How much did we contribute since 2018?

So, given that from the 8 R&D persons, we at least contribute back 1 FTE to the Drupal ecosystem. Combined with contributions that happen back from the projects by our development teams and the core contribution sponsorships for Swentel we easily get to 2 FTE's to contribute back on a total headcount of 65. 

If we add up the efforts, next to the payroll, in community work, sponsorships and memberships Dropsolid contributes an equivalent of +- 4.5% of our annual yearly spent in 2018. With the Drupalcon Amsterdam in 2019 happening as a Diamond Sponsor, this will be even bigger. We're aiming to send more than 20 payroll employees to Drupalcon and have 3 selected sessions! We're also not cutting back on our efforts to contribute back, on the contrary!

War on Talent

Being visible in the ecosystem with code contributions, event contributions, sponsorships helps to be a company people want to work for. Obviously this is not the only motivation but given that in our technical teams we employ around 1/4th of the employees as remote developers, a good company brand is important. We see that this works and this helps us to attract and retain highly skilled employees. We now have employees in countries such as Portugal, Slovenia, Poland, Ukraine, Romania, Bulgaria, ...

 

Lead, don't just follow

By actively participating, whether it is code or events or anything that decides the future of Drupal or for that matter Digital Experiences, you are ahead of the curve. This allows for more competitive offers towards clients and this helps you win deals that as a follower of the pack might not be possible. Dropsolid believes in taking the lead when it comes to being the best Drupal partner in Greater Europe, and is proud to believe we are. Up to you to believe us or not ;-)

Are you a client or looking for a job and are interested in hearing more, don't hesitate to contact us.

Nick Veenhof
Categories:

clemens-tolboom commented on pull request hechoendrupal/drupal-console#4139

On github - Thu, 2019/08/22 - 8:28pm
clemens-tolboom commented on pull request hechoendrupal/drupal-console#4139 Aug 22, 2019 clemens-tolboom commented Aug 22, 2019

@clemens-tolboom this is ready to test? ? ... we just need the semi-colon back right?

Bounteous.com: Drush 8 to Drush 9 Migration Path

Planet Drupal - Thu, 2019/08/22 - 8:28pm
With the release of Drupal 9 coming soon, the need to update form Drush 8 to Drush 9 is imminent.
Categories:

clemens-tolboom commented on issue hechoendrupal/drupal-console#4113

On github - Thu, 2019/08/22 - 1:48pm
clemens-tolboom commented on issue hechoendrupal/drupal-console#4113 Aug 22, 2019 clemens-tolboom commented Aug 22, 2019

Steps to reproduce vendor/bin/drupal --debug Drupal Console version 1.9.2 Per-Site path: /Users/clemens/Sites/drupal/d8/www/vendor/drupal/console/b…