Chromatic: Migrating (away) from the Body Field

Planet Drupal - Thu, 2016/08/18 - 7:46pm

As we move towards an ever more structured digital world of APIs, metatags, structured data, etc., and as the need for content to take on many forms across many platforms continues to grow, the humble “body” field is struggling to keep up. No longer can authors simply paste a word processing document into the body field, fix some formatting issues and call content complete. Unfortunately, that was the case for many years and consequently there is a lot of valuable data locked up in body fields all over the web. Finding tools to convert that content into useful structured data without requiring editors to manually rework countless pieces of content is essential if we are to move forward efficiently and accurately.

Here at Chromatic, we recently tackled this very problem. We leveraged the Drupal Migrate module to transform the content from unstructured body fields into re-usable entities. The following is a walkthrough.

Problem

On this particular site, thousands of articles from multiple sources were all being migrated into Drupal. Each article had a title and body field with all of the images in each piece of content embedded into the body as img tags. However, our new data model stored images as separate entities along with the associated metadata. Manually downloading all of the images, creating new image entities, and altering the image tags to point to the new image paths, clearly was not a viable or practical option. Additionally, we wanted to convert all of our images to lazy loaded images, so having programmatic control over the image markup during page rendering was going to be essential. We needed to automate these tasks during our migration.

Our Solution

Since we were already migrating content into Drupal, adapting Migrate to both migrate the content in and fully transform it all in one repeatable step was going to be the best solution. The Migrate module offers many great source classes, but none can use img elements within a string of HTML as a source. We quickly realized we would need to create a custom source class.

A quick overview of the steps we’d be taking:

  1. Building a new source class to find img tags and provide them as a migration source.
  2. Creating a migration to import all of the images found by our new source class.
  3. Constructing a callback for our content migration to translate the img tags into tokens that reference the newly created image entities.
Building the source class

Migrate source classes work by finding all potential source elements and offering them to the migration class in an iterative fashion. So we need to find all of the potential image sources and put them into an array that can used a source for a migration. Source classes also need to have a unique key for each potential source element. During a migration the getNextRow() method is repeatedly called from the parent MigrateSource class until it returns FALSE. So let's start there and work our way back to the logic that will identify the potential image sources.

** * Fetch the next row of data, returning it as an object. * * @return object|bool * An object representing the image or FALSE when there is no more data available. */ public function getNextRow() { // Since our data source isn't iterative by nature, we need to trigger our // importContent method that builds a source data array and counts the number // of source records found during the first call to this method. $this->importContent(); if ($this->matchesCurrent < $this->computeCount()) { $row = new stdClass(); // Add all of the values found in @see findMatches(). $match = array_shift(array_slice($this->matches, $this->matchesCurrent, 1)); foreach ($match as $key => $value) { $row->{$key} = $value; } // Increment the current match counter. $this->matchesCurrent++; return $row; } else { return FALSE; } }

Next let's explore our importContent() method called above. First, it verifies that it hasn't already been executed, and if it has not, it calls an additional method called buildContent().

/** * Find and parse the source data if it hasn't already been done. */ private function importContent() { if (!$this->contentImported) { // Build the content string to parse for images. $this->buildContent(); // Find the images in the string and populate the matches array. $this->findImages(); // Note that the import has been completed and does not need to be // performed again. $this->contentImported = TRUE; } }

The buildContent() method calls our contentQuery() method which allows us to define a custom database query object. This will supply us with the data to parse through. Then back in the buildContent() method we loop through the results and build the content property that will be parsed for image tags.

/** * Get all of the HTML that needs to be filtered for image tags and tokens. */ private function buildContent() { $query = $this->contentQuery(); $content = $query->execute()->fetchAll(); if (!empty($content)) { // This builds one long string for parsing that can done on long strings without // using too much memory. Here, we add fields ‘foo’ and ‘bar’ from the query. foreach ($content as $item) { $this->content .= $item->foo; $this->content .= $item->bar; } // This builds an array of content for parsing operations that need to be performed on // smaller chunks of the source data to avoid memory issues. This is is only required // if you run into parsing issues, otherwise it can be removed. $this->contentArray[] = array( 'title' => $item->post_title, 'content' => $item->post_content, 'id' => $item->id, ); } }

Now we have the the logic setup to iteratively return row data from our source. Great, but we still need to build an array of source data from a string of markup. To do that, we call our custom findImages() method from importContent(), which does some basic checks and then calls all of the image locating methods.

We found it is best to create methods for each potential source variation, as image tags often store data in multiple formats. Some examples are pre-existing tokens, full paths to CDN assets, relative paths to images, etc. Each often requires unique logic to parse properly, so separate methods makes the most sense.

/** * Finds the desired elements in the markup. */ private function findImages() { // Verify that content was found. if (empty($this->content)) { $message = 'No html content with image tags to download could be found.'; watchdog('example_migrate', $message, array(), WATCHDOG_NOTICE, 'link'); return FALSE; } // Find images where the entire source content string can be parsed at once. $this->findImageMethodOne(); // Find images where the source content must be parsed in chunks. foreach ($this->contentArray as $id => $post) { $this->findImageMethodTwo($post); } }

This example uses a regular expression to find the desired data, but you could also use PHP Simple HTML DOM Parser or the library of your choice. It should be noted that I opted for a regex example here to keep library-specific code out of my code sample. However, we would highly recommend using a DOM parsing library instead.

/** * This is an example of a image finding method. */ private function findImageMethodOne() { // Create a regex to look through the content. $matches = array(); $regex = '/regex/to/find/images/'; preg_match_all($regex, $this->content, $matches, PREG_SET_ORDER); // Set a unique row identifier from some captured pattern of the regex- // this would likely be the full path to the image. You might need to // perform cleanup on this value to standardize it, as the path // to /foo/bar/image.jpg, example.com/foo/bar/image.jpg, and // http://example.com/foo/bar/image.jpg should not create three unique // source records. Standardizing the URL is key for not just avoiding // creating duplicate source records, but the URL is also the ID value you // will use in your destination class mapping callback that looks up the // resulting image entity ID from the data it finds in the body field. $id = ‘http://example.com/foo/bar/image.jpg’; // Add to the list of matches after performing more custom logic to // find all of the correct chunks of data we need. Be sure to set // every value here that you will need when constructing your entity later. $this->matches[$id] = array( 'url' => $src, 'alt' => $alttext, 'title' => $description, 'credit' => $credit, 'id' => $id, 'filename' => $filename, 'custom_thing' => $custom_thing, ); } Importing the images

Now that we have our source class complete, let's import all of the image files into image entities.

/** * Import images. */ class ExampleImageMigration extends ExampleMigration { /** * {@inheritdoc} */ public function __construct($arguments) { parent::__construct($arguments); $this->description = t('Creates image entities.'); // Set the source. $this->source = new ExampleMigrateSourceImage(); ...

The rest of the ExampleImageMigration is available in a Gist, but it has been omitted here for brevity. It is just a standard migration class that maps the array keys we put into the matches property of the source class to the fields of our image entity.

Transforming the image tags in the body

With our image entities created and the associated migration added as a dependency, we can begin sorting out how we will convert all of the image tags to tokens. This obviously assumes you are using tokens, but hopefully this will shed light on the general approach, which can then be adapted to your specific needs.

Inside our article migration (or whatever you happen to be migrating that has the image tags in the body field) we implement the callbacks() method on the body field mapping.

// Body. $this->addFieldMapping('body', 'post_content') ->callbacks(array($this, 'replaceImageMarkup'));

Now let's explore the logic that replaces the image tags with our new entity tokens. Each of the patterns references below will likely correspond to unique methods in the ExampleMigrateSourceImage class that find images based upon unique patterns.

/** * Converts images into image tokens. * * @param string $body * The body HTML. * * @return string * The body HTML with converted image tokens. */ protected function replaceImageMarkup($body) { // Convert image tags that follow a given pattern. $body = preg_replace_callback(self::IMAGE_REGEX_FOO, `fooCallbackFunction`, $body); // Convert image tags that follow a different pattern. $body = preg_replace_callback(self::IMAGE_REGEX_BAR, `barCallbackFunction`, $body); return $body;

In the various callback functions we need to do several things:

  1. Alter the source string following the same logic we used when we constructed our potential sources in our source class. This ensures that the value passed in the $source_id variable below matches a value in the mapping table created by the image migration.
  2. Next we call the handleSourceMigration() method with the altered source value, which will find the destination id associated with the source id.
  3. We then use the returned image entity id to construct the token and replace the image markup in the body data.
$image_entity_id = self::handleSourceMigration('ExampleImageMigration', $source_id); Implementation Details

Astute observers will notice that we called self::handleSourceMigration(), not $this->handleSourceMigration. This is due to the fact that the handleSourceMigration() method defined in the Migrate class is not static and uses $this within the body of the method. Callback functions are called statically, so the reference to $this is lost. Additionally, we can't instantiate a new Migration class object to get around this, as the Migrate class is an abstract class. You also cannot pass the current Migrate object into the callback function, due to the Migrate class not supporting additional arguments for the callbacks() method.

Thus, we are stuck trying to implement a singleton or global variable that stores the current Migrate object, or duplicating the handleSourceMigration() method and making it work statically. We weren’t a fan of either option, but we went with the latter. Other ideas or reasons to choose the alternate route are welcome!

If you go the route we chose, these are the lines you should remove from the handleSourceMigration method in the Migrate class when you duplicate it into one of your custom classes.

- // If no destination ID was found, give each source migration a chance to - // create a stub. - if (!$destids) { - foreach ($source_migrations as $source_migration) { - // Is this a self reference? - if ($source_migration->machineName == $this->machineName) { - if (!array_diff($source_key, $this->currentSourceKey())) { - $destids = array(); - $this->needsUpdate = MigrateMap::STATUS_NEEDS_UPDATE; - break; - } - } - // Break out of the loop if a stub was successfully created. - if ($destids = $source_migration->createStubWrapper($source_key, $migration)) { - break; - } - } - }

Before we continue, let's do a quick recap of the steps along the way.

  1. We made an iterative source of all images from a source data string by creating the ExampleMigrateSourceImage class that extends the MigrateSource class.
  2. We then used ExampleMigrateSourceImage as a migration source class the in the ExampleImageMigration class to import all of the images as new structured entities.
  3. Finally, we built our "actual" content migration and used the callbacks() method on the body field mapping in conjunction with the handleSourceMigration() method to convert the existing image markup to entity based tokens.
The end result

With all of this in place, you simply sit back and watch your migrations run! Of course before that, you get the joy of running it countless times and facing edge cases with malformed image paths, broken markup, new image sources you were never told about, etc. Then at the end of the day you are left with shiny new image entities full of metadata that are able to be searched, sorted, filtered, and re-used! Thanks to token rendering (if you go that route), you also gain full control over how your img tags are rendered, which greatly simplifies the implementation of lazy-loading or responsive images. Most importantly, you have applied structure to your data, and you are now ready to transform and adapt your content as needed for any challenge that is thrown your way!

Categories:

Jeff Geerling's Blog: Increase the Guzzle HTTP Client request timeout in Drupal 8

Planet Drupal - Thu, 2016/08/18 - 6:56pm

During some migration operations on a Drupal 8 site, I needed to make an HTTP request that took > 30 seconds to return all the data... and when I ran the migration, I'd end up with exceptions like:

Migration failed with source plugin exception: Error message: cURL error 28: Operation timed out after 29992 milliseconds with 2031262 out of 2262702 bytes received (see http://curl.haxx.se/libcurl/c/libcurl-errors.html).

The solution, it turns out, is pretty simple! Drupal's \Drupal\Core\Http\ClientFactory is the default way that plugins like Migrate's HTTP fetching plugin get a Guzzle client to make HTTP requests (though you could swap things out if you want via services.yml), and in the code for that factory, there's a line after the defaults (where the 'timeout' => 30 is defined) like:

Categories:

Drupal core announcements: We can add big new things to Drupal 8, but how do we decide what to add?

Planet Drupal - Thu, 2016/08/18 - 2:48pm

Drupal 8 introduced the use of Semantic Versioning, which practically means the use of three levels of version numbers. The current release is Drupal 8.1.8. While increments of the last number are done for bugfixes, the middle number is incremented when we add new features in a backwards compatible way. That allows us to add big new things to Drupal 8 while it is still compatible with all your themes and modules. We successfully added some new modules like BigPipe, Place Block, etc.

But how do we decide what will get in core? Should people come up with ideas, build them, and once they are done, they are either added in core or not? No. Looking for feedback at the end is a huge waste of time, because maybe the idea is not a good fit for core, or it clashes with another improvement in the works. But how would one go about getting feedback earlier?

We held two well attended core conversations at the last DrupalCon in New Orleans titled The potential in Drupal 8.x and how to realize it and Approaches for UX changes big and small both of which discussed a more agile approach to avoid wasting time.

The proposal is to separate the ideation and prototyping process from implementation. Within the implementation section the potential use of experimental modules helps with making the perfecting process more granular for modules. We are already actively using that approach for implementation. On the other hand the ideation process is still to be better defined. That is where we need your feedback now.

See https://www.drupal.org/node/2785735 for the issue to discuss this. Looking forward to your feedback there.

Categories:

Mediacurrent: How Drupal won an SEO game without really trying

Planet Drupal - Thu, 2016/08/18 - 2:23pm

At Mediacurrent we architected and built a Drupal site for a department of a prominent U.S. university several years ago. As part of maintaining and supporting the site over the years, we have observed how well it has performed in search engine rankings, often out-performing other sites across campus built on other platforms.

Categories:

KnackForge: Drupal Commerce - PayPal payment was successful but order not completed

Planet Drupal - Thu, 2016/08/18 - 12:00pm
Drupal Commerce - PayPal payment was successful but order not completed

Most of us use PayPal as a payment gateway for our eCommerce sites. Zero upfront, No maintenance fee, API availability and documentation makes anyone easy to get started. At times online references offer out-dated documentation or doesn't apply to us due to account type (Business / Individual), Country of the account holder, etc. We had this tough time when we wanted to set up Auto return to Drupal website.

Thu, 08/18/2016 - 15:30 Tag(s) Drupal planet Drupal 7 DropThemes.in drupal-commerce
Categories:

Unimity Solutions Drupal Blog: Video Annotations: A Powerful and Innovative Tool for Education

Planet Drupal - Thu, 2016/08/18 - 8:51am

According to John J Medina a famous molecular biologist “Vision trumps all other senses.” Human mind is attracted to remember dynamic pictures rather than listen to words or read long texts. Advancement in multimedia has enabled teachers to impart visual representation of content in the class room.

Categories:

Drupalize.Me: Learn by Mentoring at DrupalCon

Planet Drupal - Thu, 2016/08/18 - 8:37am

DrupalCon is a great opportunity to learn all kinds of new skills and grow professionally. For the 3 days of the main conference in Dublin (September 27–29) there will be sessions on just about everything related to Drupal that you could want. One amazing opportunity that you may not be aware of though is the Mentored Sprint on Friday, September 30th. This is a great place for new folks to learn the ropes of our community and how to contribute back. What may be less talked about is the chance to be a mentor.

Categories:

Roy Scholten: Vetting Drupal product ideas

Planet Drupal - Wed, 2016/08/17 - 11:57pm

We’ve made big strides since Drupalcon New Orleans in how we add new features to Drupal core. The concept of experimental modules has already helped us introduce features like a new way to add blocks to a page, content moderation and workflow tools and a whole new approach for editing all the things on a page while staying at that page.

In New Orleans we started to define the process for making these kinds of big changes. Probably the most important and defining aspect of it is that we’re (finally!) enabling a clearer separation between vetting ideas first, implementation second.

True to form we specified and detailed the latter part first :-)

So, on to that first part, vetting Drupal product ideas. In my core conversation I outlined the need for making bigger UX changes, faster and suggested possible approaches for how to design and develop those, borrowing heavily from the Lean UX method

Since then, we’ve been reminded that we really do need a clearly defined space to discuss the strategic value of proposed new features. A place to decide if a given idea is desirable and viable as an addition to core.

The point being: core product manager with help from Drupal UX team members wrote up a proposal for how to propose core product ideas and what’s needed to turn a good idea into an actionable plan.

It needs your feedback. Please read and share your thoughts.

Tags: drupaluxprocessdrupalplanetSub title: Agree on why and what before figuring out the how
Categories:

Mediacurrent: DrupalCon NOLA: The People Behind the Usernames

Planet Drupal - Wed, 2016/08/17 - 8:33pm

As we work every day on our own projects, with our own deadlines and priorities, it is often too easy to forget about the entire community of others using Drupal in much the same way. When we're working with Drupal in our various capacities, there is no shortage of methods to interact with the community and contribute back, but those aren't the focus of this post.

Categories:

myDropWizard.com: Drupal 6 security updates for Panels!

Planet Drupal - Wed, 2016/08/17 - 8:16pm

As you may know, Drupal 6 has reached End-of-Life (EOL) which means the Drupal Security Team is no longer doing Security Advisories or working on security patches for Drupal 6 core or contrib modules - but the Drupal 6 LTS vendors are and we're one of them!

Today, there is a Critical security releases for the Panels modules for multiple Access Bypass vulnerabilities.

The first vulnerability allows anonymous users to use AJAX callbacks to pull content and configuration from Panels, which allow them to access private data. And the second, allows authenticated users with permission to use the Panels IPE to modify the Panels display for pages that they don't have permission to otherwise edit.

See the security advisory for Drupal 7 for more information.

Here you can download the patch for 6.x-3.x!

If you have a Drupal 6 site using the Panels module, we recommend you update immediately! We have already deployed the patch for all of our Drupal 6 Long-Term Support clients. :-)

If you'd like all your Drupal 6 modules to receive security updates and have the fixes deployed the same day they're released, please check out our D6LTS plans.

Note: if you use the myDropWizard module (totally free!), you'll be alerted to these and any future security updates, and will be able to use drush to install them (even though they won't necessarily have a release on Drupal.org).

Categories:

Rakesh's DSoC 16 blog: Last week - GSoC16 - Detected and Solved Merge Conflicts in Drupal8.

Planet Drupal - Wed, 2016/08/17 - 8:04pm
Last week - GSoC16 - Detected and Solved Merge Conflicts in Drupal8. rakesh Wed, 08/17/2016 - 23:34
Categories:

Miloš Bovan: The final week of Google Summer of Code 2016

Planet Drupal - Wed, 2016/08/17 - 7:56pm
The final week of Google Summer of Code 2016

It’s been more than 13 warm weeks since I’ve started to work on my Google Summer of Code 2016 - Mailhandler project. During the past few weeks, I finished the coding and worked on latest improvements related to documentation.

In the past week, the project documentation has been updated which was the last step before merging Github repository and Sandbox project into Mailhandler. Drupal 8 release section was updated and it summarizes the features of D8 version. If you are wondering where you can use the code I developed this summer, feel free to read about real Mailhanlder use-cases.

This week, I am doing the latest cleanups and preparing the project for the final evaluation. It is the second evaluation after I submitted the midterm evaluation. In case you missed the previous posts from Google Summer of Code series, I am providing the project retrospect below.
 

Project Retrospect

In early February, in an ordinary conversation, a friend of mine told me about this year’s Google Summer of Code program. I got interested in it since I took apart in GHOP (The Google Highly Open Participation Contest; This program was replaced with Google Code-In) during high-school.

In March, I’ve found out that Drupal will be one of the participating organizations and since I did a Drupal internship last year, this seemed to be a perfect opportunity to continue open-source contribution.

Among many interesting project ideas, I decided for “Port Mailhandler to Drupal 8”. Miro Dietiker proposed the project and during the proposal discussions, Primoz Hmeljak joined the mentoring team too.

The great news came in April. I was one of the 11 students chosen to contribute Drupal this summer! First Yay moment!

The project progress could have been followed on this blog, so I’m will not go deeper into it.

3 months of work passed quickly and at this point, I can confirm that I learned a ton of new things. I improved not only my coding skills but communication and project management skills too.

In my opinion, we reached all the goals we put on the paper in April. All the features defined in the proposal were developed, tested and documented.

Google Summer of Code is a great program and I would sincerely recommend all the students to consider participating in it.

Future plans

The future plans about the module are related to its maintenance. Inmail is still under development and it seems it will be ready for an alpha release very soon. In the following days, I am going to create an issue about nice-to-have features of Mailhandler. This issue could serve as a place for Drupal community to discuss the possible ways of the future Mailhandler development.


Thank you note

Last but not least, I would like to give huge thanks to my mentors (Miro Dietiker and Primoz Hmeljak) for being so supportive, helpful and flexible during this summer. Their mentoring helped me in many ways - from the proposal re-definition in April/May, endless code iterations, discussions of different ideas to blog post reviews, including this one :).

I would like to thank my company too for allowing me to contribute Drupal via this program and Google for giving me an opportunity to participate in this great program.

 

 

Milos Wed, 08/17/2016 - 19:56 Tags Drupal Google Summer of Code Open source Drupal Planet Add new comment
Categories:

lakshminp.com: Tagged services in Drupal 8

Planet Drupal - Wed, 2016/08/17 - 4:07pm

We saw how to write a simple service container in Drupal earlier. We shall build a tagged service now. To illustrate a proper use case for tagged services, let's contrive a scenario where you add a pipeline custom filters to user text before rendering it on the page.

First, clone the code which will be used in this post.

$ git clone git@github.com:drupal8book/process_text.git

Checkout the first version of the code where we take custom text from user, process and display it in a page without using services.

$ cd process_text $ git checkout -f just-filtering

We get custom text from a user using a config form.

class CustomTextSettingForm extends ConfigFormBase { /** * {@inheritdoc} */ protected function getEditableConfigNames() { return [ 'process_text.settings', ]; } /** * {@inheritdoc} */ public function getFormId() { return 'custom_text_setting_form'; } /** * {@inheritdoc} */ public function buildForm(array $form, FormStateInterface $form_state) { $config = $this->config('process_text.settings'); $form['custom_text'] = [ '#type' => 'textarea', '#title' => $this->t('Custom Text'), '#default_value' => $config->get('custom_text'), ]; return parent::buildForm($form, $form_state); } /** * {@inheritdoc} */ public function validateForm(array &$form, FormStateInterface $form_state) { parent::validateForm($form, $form_state); } /** * {@inheritdoc} */ public function submitForm(array &$form, FormStateInterface $form_state) { parent::submitForm($form, $form_state); $this->config('process_text.settings') ->set('custom_text', $form_state->getValue('custom_text')) ->save(); $form_state->setRedirect('process_text.show'); } }

We save it as a piece of configuration called process_text.settings.custom_text.

Before rendering this text, let's say you would want to:

  • Remove any <div> tags.
  • Substitute a token [greeting] with <span class"greeting">hello world</span> throughout the text.

We get the text and do all the above processing inside a custom controller.

class ProcessTextController extends ControllerBase { /** * Processtext. * * @return string * Return processed custom text. */ public function processText() { $custom_text = \Drupal::config('process_text.settings')->get('custom_text'); // do processing // remove divs $custom_text = str_replace(["<div>", "</div>"], "", $custom_text); // replace greeting tokens $custom_text = str_replace("[greeting]", '<span class="greeting">hello world</span>', $custom_text); return [ '#type' => 'markup', '#markup' => $custom_text ]; } }

This is good, but we could do better. What if we change the filter applying mechanism? We have to change this code. Instead, let's convert it into a service.

$ cd process_text $ git checkout -f services-first-cut

Our text filter service takes a set of filters and applies them to a given text when we call applyFilters.

class TextFilterService { private $filters = []; /** * @param Filter $filter */ public function addFilter(Filter $filter) { $this->filters[] = $filter; } /** * applies all filters to given text and returns * filtered text. * * @param string $txt * * @return string */ public function applyFilters($txt) { foreach ($this->filters as $filter) { $txt = $filter->apply_filter($txt); } return $txt; } }

We need to crate a services.yml file for the above service.

services: process_text.text_filter: class: Drupal\process_text\TextFilterService

Here's how the processText function text looks now.

public function processText() { $custom_text = \Drupal::config('process_text.settings')->get('custom_text'); // do processing using a service $filter_service = \Drupal::service('process_text.text_filter'); // remove divs $filter_service->addFilter(new RemoveDivs()); // substitute greeting token $filter_service->addFilter(new Greeting()); // apply all the above filters $custom_text = $filter_service->applyFilters($custom_text); return [ '#type' => 'markup', '#markup' => $custom_text ]; }

Now the filter applying mechanism is swappable. We can add write a different functionality and inject that implementation using service containers.

Now, what if we want to add a new filter to this code, like, enclosing the whole text within a <p> tag.

Sure. We could do that.

Let's checkout the specific tag where we add a new filter.

$ cd process_text $ git checkout -f add-new-filter

We build that filter.

class EnclosePTags implements Filter { public function apply_filter($txt) { return '<p>'. $txt . '</p>'; } }

…and add it to the set of filters being applied.

public function processText() { $custom_text = \Drupal::config('process_text.settings')->get('custom_text'); // do processing using a service $filter_service = \Drupal::service('process_text.text_filter'); // remove divs $filter_service->addFilter(new RemoveDivs()); // substitute greeting token $filter_service->addFilter(new Greeting()); // Enclose p tags $filter_service->addFilter(new EnclosePTags()); // apply all the above filters $custom_text = $filter_service->applyFilters($custom_text); return [ '#type' => 'markup', '#markup' => $custom_text ]; }

How about injecting the filter adding mechanism itself? Wouldn't it be cool if we are able to add new filters without changing this part of the code? Not to mention the fact that the code will be more testable than before if we follow this approach. This is exactly what tagged services help us accomplish.

Let's write each filter as a tagged service.

$ cd process_text $ git checkout -f tagged-services

Here's how our process_text.services.yml looks now.

services: process_text.text_filter: class: Drupal\process_text\TextFilterService tags: - { name: service_collector, tag: text_filter, call: addFilter } remove_divs_filter: class: Drupal\process_text\TextFilter\RemoveDivs tags: - { name: text_filter } greeting_filter: class: Drupal\process_text\TextFilter\Greeting tags: - { name: text_filter } enclose_p_filter: class: Drupal\process_text\TextFilter\EnclosePTags tags: - { name: text_filter }

There are many changes here. Firstly, all the filters have been converted to services themselves. The have a common tag called text_filter. The main service also has a few changes. It has a tag called service_collector and a tag parameter call. This ritual of creating a service container and adding a set of tagged services is such a common pattern that Drupal 8 has a special tag to do this, called the service_collector. This tag takes an additional parameter called call which indicates what function has to be called in the service to add all the tagged services.

What happens is, Drupal's TaggedHandlersPass picks up all services with "service_collector" tag, finds services which have the same tag as that of this service(text_filter in our case) and calls the method in call to consume the tagged service definition. If you're coming from Symfony world, this might seem familiar for you. In order to execute some custom code, like applying a set of filters, we implement CompilerPassInterface, which is run whenever the service cotainer(ApplyFilter in our case) is being built. You can find more about CompilerPassInterface here.

Your controller code looks a lot simpler now.

public function processText() { $custom_text = \Drupal::config('process_text.settings')->get('custom_text'); // do processing using a service $filter_service = \Drupal::service('process_text.text_filter'); $custom_text = $filter_service->applyFilters($custom_text); return [ '#type' => 'markup', '#markup' => $custom_text ]; }

Now, all you need to add new filters is to update the service yaml file with the new filter service and tag it with "text_filter" tag.

Tagged services in the wild

Drupal allows developers to add a new authentication mechanism using tagged services. The authentication_collector is defined in core.services.yml.

authentication_collector: class: Drupal\Core\Authentication\AuthenticationCollector tags: - { name: service_collector, tag: authentication_provider, call: addProvider }

To add a new authentication provider, one has to implement the AuthenticationProviderInterface and flesh out the applies and authenticate functions. This will be the subject of another post.

Here's how the addProvider function looks like:

public function addProvider(AuthenticationProviderInterface $provider, $provider_id, $priority = 0, $global = FALSE) { $this->providers[$provider_id] = $provider; $this->providerOrders[$priority][$provider_id] = $provider; // Force the providers to be re-sorted. $this->sortedProviders = NULL; if ($global) { $this->globalProviders[$provider_id] = TRUE; } }

And here's how we would register our hypothetical authentication provider service.

services: authentication.custom_auth: class: Drupal\custom_auth\Authentication\Provider\CustomAuth tags: - { name: authentication_provider }

Another example is the breadcrumb manager service.

breadcrumb: class: Drupal\Core\Breadcrumb\BreadcrumbManager arguments: ['@module_handler'] tags: - { name: service_collector, tag: breadcrumb_builder, call: addBuilder }

To add breadcrumbs from your module, you would need to implement BreadcrumbBuilderInterface and add the following entry to your services.yml,

services: foo.breadcrumb: class: Drupal\foo\MyBreadcrumbBuilder tags: - { name: breadcrumb_builder, priority: 100 }

The BreadcrumbManager::addBuilder collects all breadcrumb bilders and builds it using the BreadcrumbManager::build function.

public function addBuilder(BreadcrumbBuilderInterface $builder, $priority) { $this->builders[$priority][] = $builder; // Force the builders to be re-sorted. $this->sortedBuilders = NULL; } Drupal, Drupal 8, Drupal Planet
Categories:

Lullabot: Making Web Accessibility Great Again: Auditing the US Presidential Candidates Websites for Accessibility

Planet Drupal - Wed, 2016/08/17 - 3:00pm

Imagine that you arrive at a website, but you cannot see the screen. How do you know what’s there? How do you navigate? This is normal for many people, and the accessibility of your site will make or break their experience. Accessibility is about including everyone. People with physical and cognitive disabilities have specific challenges online—and making your site accessible removes those barriers and opens the door to more users.

Severely disabled Americans constitute a population of 38.3 million people, and make up a huge swath of voters (see the #CripTheVote movement on Twitter). Some notable U.S. presidential elections have been decided by much less, and because of this, we’re auditing the US presidential election candidates’ websites.

During this audit, we’ll see what the candidates’ websites are doing right and wrong, and where the low-hanging fruit lies. This article won’t be a full top-to-bottom audit, but we will show some of the important things to look for and explain why they’re important.

Our Methods Automated Testing

The first step in our a11y audit is to do a quick automated check. If you’re new to accessibility, the WAVE tool by WebAIM is a great place to start. It’ll check for standard accessibility features and errors in alt attributes, contrast, document outline, and form labels. For the features or errors it finds, it provides an info icon that you can click to learn what the issue is, why it’s important, and how to do it right. WAVE is free, and highlights both negative (errors, alerts, features, structural elements, ARIA attributes, contrast), and positive (features, structural elements, ARIA attributes).

Keyboard Testing

As great as WAVE is, an automated tool is never as good as a live person. This is because some accessibility requirements need human logic to apply them properly. For the next part, we’re going to navigate around each website using only the keyboard.

This is done by using the tab button to move to the next element, and shift-tab to move backwards. The spacebar (or return) is used to click or submit a form element. If everything is done right, a person will be able to navigate through your website without falling into tab rabbit-holes (tabbit-holes?). We should be able to tab through the whole page in a logical order without getting stuck or finding things that we can’t access or interact with.

Beyond that, we need to be able to see where the focus lies as we tab across the page. Just as interactive elements give a visual cue on hover, we should get an indication when we land on an interactive element while tabbing, too. That state is referred to as ‘having focus’. You can extend your hover state to focus, or you can make a whole new interaction for focus. It’s up to you!

.link--cta-button:hover, .link--cta-button:focus { /* The :focus pseudo-class for a11y */ background: #2284c0; } Screen Reader Audit

Screen readers are used by visually impaired and blind people to navigate websites. For this purpose we’ll use VoiceOver, which is the default pre-installed screen reader for OS X. We’re looking for things that read oddly (like an acronym), things that don’t get read that should get read (and vice-versa), and making sure all of the information is available.

Let’s start with Donald Trump’s website

The first thing that we did while looking at Donald Trump’s website was audit it using the WAVE tool. Here’s what we found:

  • 8 Errors
  • 14 Alerts
  • 15 Features
  • 35 Structural Elements
  • 5 HTML5 and ARIA
  • 5 Contrast Errors (2 were false flags)
The Good Bold Colors for a Bold Candidate

The color scheme is very accessible. On the front page, there were only two places where the contrast wasn’t sufficient. Outside of that, his color scheme provides text that stands out well from the background and is easy to read for low-vision users.

The Bad Lacking Focus

Remember how we talked about focus states? This site has almost none. Tabbing through the page is confusing because there are no visual indications of where you are.

This is especially egregious because the browser automatically applies focus states to focusable elements for you. In order for there not to be focus elements at all, the developer has to actively go out of their way to break them by applying outline: 0; to the element’s focus state. This is okay if you’re doing it to replace the focus state with something more decorative, but taking it off and not replacing it is a big accessibility no-no.

Skipped Skip Link

When tabbing through The Donald’s website, the first thing we notice is the absence of a skip link. Without a skip link, a keyboard user is forced to tab through each link in the navigation when arriving at each page before they can access the rest of the content. This repetitive task can become aggravating quickly, especially on sites with a lot of navigation links.

Unclear Link Text

Links should always have text that clearly explains where the user is going. The link’s text is what a person using a screen reader will hear, so text like ‘Read More’ or text and icons that require visual context to understand their destination aren’t ideal.

In this area, the link text that goes out to the linked Twitter posts read ‘12h’ and ‘13h’. Without the visual context of a Twitter icon (that’s a background image, so there’s no alternative text to provide that), the user probably has no idea what ‘12h’ is referring to or where that link will lead.

The Ugly Navigation Nightmares

The most important part of any website, in terms of access, is the navigation. An inaccessible navigation menu can block access to large portions of the website. Unfortunately, the navigation system of Trump’s website does just that, and prevents users with disabilities from directly accessing the sub-navigation items under the issues and media sections.

A more accessible way to do this is to use a click dropdown instead of a :hover. If that doesn’t work for the design, make sure that the :hover state of the menu applies to :focus as well, so that the menu will open the nested links when the parent menu item is tabbed to.

Disorganized Structure

Structural elements (h1, h2, h3, etc tags) are very helpful when used properly. In this case, they’re definitely not. Heading levels aren’t sequential, and nested information isn’t always relevant to its parent.

Audit of Hillary Clinton’s website The Good

Overall, Clinton’s website is better than most when it comes to accessibility. It’s that clear that her development team made it a purposeful consideration during the development process. While it’s not perfect, there was a lot of good done here. Let’s explore some examples of things done right.

Keyboard Accessibility

The keyboard accessibility on this site is very good. We found that we could access the elements and navigate to other pages easily without a mouse. It was easy to open and shut the drop-down ‘More’ area in the navigation, and access its nested links, which is a good example of how to implement what we were talking about when we covered the shortfalls of the navigation system on Trump’s website.

Skip Link

Hillary Clinton’s website includes a proper skip link, which allows users to skip the navigation and go directly to the content.

Great Focus States

The other thing we found when checking the keyboard accessibility was that everything has a focus state that makes it visually obvious where you are on the page. The light dotted border focus state is a bit subtle for low-vision users, but the fact that the focus state of the elements was styled independently from the hover state shows that the developer was aware of the need for focus indicators and made a conscious effort to implement them.

Translation

We usually think of accessibility in terms of people with disabilities because they often benefit from it the most, but accessibility is really just about including as many people as possible. A nice touch we found on Clinton’s site was a button at the top to translate the site into Spanish. With 41 million native Spanish speakers in the US, providing the option to experience the content in the user’s first language is a great accessibility move.

Video Captioning

Deaf people rely on captions to get the dialogue from videos, since it’s very difficult to lip-read in film. The videos on Hillary’s site are furnished with open captions, which means that they’re always on. Open captions are great for people with disabilities, but they’re also a smart move to capture your non-disabled audience as well. Often autoplay videos won’t play any sound unless they’re interacted with, but providing open captions on the video gives you another chance to capture the audience’s interest by displaying the words on the screen.

The Bad No Transcripts for Video

While it was great that the videos were captioned, we couldn’t find a transcript provided. Many people erroneously believe that you only need one or the other, but captions and transcripts actually serve different purposes, so it’s ideal to provide both. Captions are great for the Deaf, who want to read the words in real-time with the video. Transcripts are more useful for the blind and the Deaf-blind, who benefit from a written summary of what’s visually happening onscreen in each scene before the written dialogue begins. A braille terminal, used by the Deaf-blind, can’t convert open captions inlaid into the video’s frames into braille for its users, so these users won’t benefit from that.

Low Contrast

Contrast is important for low-vision users. We know that subtlety is all the rage, but design choices like putting blue text on a blue background makes it really difficult for some people to read. There are some great free tools like Das Plankton that will let you see if your contrast level is high enough to meet accessibility standards.

Schemes like this fail contrast testing on every level, so it’s best to avoid them. A better choice probably would have been white over a slightly darker blue.

The Ugly The Horrible Modal

Despite the obvious hard work that went into making Hillary’s website accessible, much of the effort is lost due to a modal that appears when visiting the website for the first time (or in incognito mode). The problem is that the modal doesn’t receive focus when it pops up, and its close button has no focus indicator. While it technically can be closed via keyboard by navigating backwards (or tabbing through every single link on the page) once it pops up, it’s not obvious visually when that close button has focus, and navigating backwards isn’t exactly intuitive.

Conclusion

With one glaring exception, it’s obvious that lots of thought and work had been put into making Hillary Clinton’s website accessible to voters with disabilities. There is definitely room for improvement with small things like somewhat irrelevant alternative attributes on photos, but on the whole the site is better on accessibility than the vast majority of the sites that we see.

Unfortunately, it is also obvious that accessibility is deeply neglected within Donald Trump’s website, which leaves a large swath of potential voters unable to browse to his stance on issues and other content. Hopefully, this will be attended to shortly.

Hopefully these auditing case studies lead you to think about your own website from the point of view of a person with a disability. There are plenty of challenges online for the disability community, but lots of those can be fixed with a few easy tweaks like the ones we covered here. We hope you'll use what you've learned to make your website accessible, too.

Categories:

Anexus: How to use ES6 in Drupal 8

Planet Drupal - Wed, 2016/08/17 - 12:58pm
How to use ES6 in Drupal 8

I bet many of you doesn't understand what is ES6, being honest I didn't know this term until a few days ago. 

ES6 stands for ECMAScript v6 or ES2015, but in simple words, is just Javascript; Correct Javascript as simple as that.

So, ES6 one of the latest version of Javascript specification, but is not supported by all browsers, because was released in June 2015, even worst in June 2016 we got ES7 and as you can imagine not supported yet.

Right now all browsers support ES5, then, if it's not compatible with all browsers, why ES6+ is a big thing? Because, now Javascript is more OOP, using classes, constants, and modules, enabling to create more complex and organized projects.

So, how we could use ES6 in our web projects like Drupal, to avoid to have a particular version of our site based on our client's browser? The solution is Babel

Babel transforms your code to ES5 to be used in your web applications. 

Babel works using plugins that allow the transformation and enable to use a particular plugin based on the source of your code like React that requires an especial plugin to do the conversion.

Let me show an example that creates a simple ReactJS.

1. Install Node.js

The first thing we need to do is install Node.js in your system, the easy way to do that is downloading the proper installer for our platform from https://nodejs.org/en/download

We need to install Node.js because we are going to use the NPM which is a package manager for javascript libraries 

2.  Modify gitignore file

Drupal 8 projects include their own .gitignore file, but because we are about to include npm in our development process, we need to add some extra rules listed below:

# Node.js # Logs logs *.log npm-debug.log* # Runtime data pids *.pid *.seed *.pid.lock # Directory for instrumented libs generated by jscoverage/JSCover lib-cov # Coverage directory used by tools like istanbul coverage # nyc test coverage .nyc_output # Grunt intermediate storage (http://gruntjs.com/creating-plugins#storing-task-files) .grunt # node-waf configuration .lock-wscript # Compiled binary addons (http://nodejs.org/api/addons.html) build/Release # Dependency directories node_modules jspm_packages # Optional npm cache directory .npm # Optional REPL history .node_repl_history
3. Create a package.json file

To be able to distribute later your javascript library you need to create a package.json file, you create that file in interactive mode using the command:

$ npm init

In the end, you will get a file similar to the following file.

{   "name": "drupalreactform",   "version": "1.0.0",   "description": "ReactJS form to be embed in Drupal 8 Controller",   "main": "index.js",   "scripts": {     "test": "echo \"Error: no test specified\" && exit 1"   }   "author": "enzo - Eduardo Garcia",   "license": "ISC" } 4. Install Webpack

To expose our library in Drupal we need to create a distribution package, for that purpose, we will use [Webpack](webpack.github.io) that is a module bundler which takes modules with dependencies to generates static assets by bundling them together.

$ npm install webpack --save

The command about will install webpack with all required libraries and modify our package.json, in the same way that we use in Composer the composer.json file.

5. Configuring Webpack & creating bundle

It's necessary to inform to Webpack using file webpack.config.js, how we can to create our bundle to be used Drupal, the following configuration assumes we have a custom module located in web/modules/custom/mysearch.

var webpack = require('webpack'); var path = require('path'); var MODULE_BUILD_DIR = path.resolve(__dirname, 'web/modules/custom/mysearch/js'); var MODULE_APP_DIR = path.resolve(__dirname, 'web/modules/custom/mysearch/js/es6'); var config = {   entry: MODULE_APP_DIR + '/mysearch.form.jsx',   output: {     path: MODULE_BUILD_DIR,     filename: 'mysearch.form.js'   } }; module.exports = config;

With the configuration above, we are saying we will load the file mysearch.form.jsx and all included files in mysearch.form.js file

If you write something simple like

console.log('Hello ES6!');

You don't need any special transformation and you can create the bundle, for that propose you need to execute the following command:

$ ./node_modules/.bin/webpack -d

You will get an output similar to this image:

The generation will work correctly because the source in ES5 and the output too; So, no transformation was required.

6. Testing transformation

I know I said we would embed the file generated in Drupal, but in a development process is faster if we could test first outside Drupal, for that, we could create a file named test/es62es5/index.htm inside module directory  with the following content.

          Testing transformation ES6 -> ES5              

Opening that file in our browser would enable any possible error and reduce the change to blame Drupal 8 for a malfunction in our code.

7. Use Babel to transform ES6

Now we need to install Babel and Babel loaders to be able to transform our ReactJS form into ES5; the next command installs the required packages.

$ npm install babel-loader babel-preset-es2015 babel-preset-react --save

Also, we need to create a .babelrc file, to inform to Babel what presents will be used in transformation, check and example of that file below:

{   "presets" : ["es2015", "react"] }

Finally, we need to modify out webpack configuration to report what loader we are going to use in our transformation, the new aspect of webpack.config.js will be like this:

var webpack = require('webpack'); var path = require('path'); var MODULE_BUILD_DIR = path.resolve(__dirname, 'web/modules/custom/mysearch/js'); var MODULE_APP_DIR = path.resolve(__dirname, 'web/modules/custom/mysearch/js/es6'); var config = {     entry: MODULE_APP_DIR + '/mysearch.form.jsx',     output: {         path: MODULE_BUILD_DIR,         filename: 'mysearch.form.js'     },     module : {         loaders : [             {                 test : /\.jsx?/,                 include : MODULE_APP_DIR,                 loader : 'babel'             }         ]     } }; 8. Create React form

Before to create the form we need to install some libraries to build our form.

$ npm install react react-dom antd --save

If we plan to embed CSS or LESS files in our app, need to install loaders for that using the following instructions, and register the loader in webpack

$ npm install css-loader less less-loader style-loader  --save-dev

The code of our form will be an example form React created in a previous blog post:

import React, { PropTypes } from 'react'; var Search = React.createClass({     render: function(){         return (             React.createElement('form', {onSubmit: this.onSubmit, className: 'SearchForm', noValidate: true},                 React.createElement('input', {                     type: 'text',                     placeholder: 'Search'                 }),                 React.createElement("select", { placeholder: 'Category', value: '', onChange: this.changeHandler },                     React.createElement("option", { value: 1 }, "Software"),                     React.createElement("option", { value: 2 }, "Movie")                 ),                 React.createElement('button', {type: 'submit'}, "Go")             )         );     }, }); ReactDOM.render(React.createElement(Search),  document.getElementById("app"));

Of course, you can create a more advanced form, importing other libraries.

9. Include form in a Drupal controller

After "compile" our form in one file, the remaining step is to include it in a Drupal Controller, to do that you just need to follow the blog entry www.anexusit.com/blog/how-to-load-js-and-css-libraries-a-drupal-8-controller.

I hope did you find this blog entry useful.
 

enzo Wed, 08/17/2016 - 04:58
Categories:

Liip: Drupalaton 2016

Planet Drupal - Wed, 2016/08/17 - 11:13am

Last week Drupalaton 2016 took place. With about 150 registrations this was the largest Drupalaton so far. The organizers did an amazing job in coping with this mass. There were two session threads and a sprint room. Of the many interesting presentations I would like to mention Fabian Bircher’s “Configuration Management: theory and practice” (a must for everyone who gets lost while trying to work in a team on a Drupal8 project) , Pieter Frenssen’s “Working with REST APIs”  (it was good to see how simple it is in Drupal8) and “Drupal 8 Media” from Pónya Péter, Szanyi Tamás and Rubén Teijeiro (seems we have a huge step forward in media handling since Drupal7!). I held a session on caching in Drupal 8 which was the shortened version the one I did on Drupal Developer Days in Milan.

Liip was a silver sponsor of the event.

Finally, some pictures on the Friday ship cruise. Thanks to Brainsum for sponsoring it!

Categories:

DrupalCon News: Build your DrupalCon agenda: the session and BOF schedule is live

Planet Drupal - Wed, 2016/08/17 - 5:00am

DrupalCon starts in just 40 days. But you can plan your schedule today. There'll be more than 140 sessions, three keynotes, and one unforgettable pre-note.

Categories:

Anchal: GSoC'16 - Port Comment Alter Module - Week 12

Planet Drupal - Wed, 2016/08/17 - 2:00am

As part of GSoC’16 I’m working on Porting Comment Alter module to Drupal 8 under the mentorship of boobaa and czigor. This blog is an excerpt of the work which I did in the twelfth week of the coding period of GSoC’16. The blogpost for the work done in the eleventh week can be found here.

The GSoC coding period ends this week and I’m glad that I was able to complete the project as per my proposal. All these three months have been very productive for me during which I learned a ton of things not just related to Drupal but also things like writing blogs, managing the project, etc.

During the coding period being stuck on a problem for 3-4 days and after countless hours of debugging finding the solution to be a one-liner or getting good reviews from mentors were some of the most intriguing and satisfying moments for me. I would like to thank both of my mentors boobaa and czigor for always helping me with coding, blogs and project management. It would have never been possible without them. I would also like to thank the whole Drupal community and Google Summer of Code for providing this opportunity to me.

Now talking about the work which I did this week, since most of the coding part was completed last week this week I focused on adding documentation for the module, doing cleaning and proof reading of the whole repo. I’ve created a short demo video: Comment Alter module for Drupal 8 demonstrating a simple use case and working of the module.


This week I also fixed some minor issues like changing the #parent property for the comment form for all the fields to avoid any clashes. In order to allow same name fields on both comment as well as parent entity the #parent property was used in the comment_alter_form_comment_form_alter() function. Due to this the submitted field values appeared at a different location preventing instances of multiple field value in the same #parent space or $form element (see commit).

In the upcoming week I would first move the module to drupal.org and will update the description about the module on the project page. I planned to move the module to drupal.org last week only but because of some complications, my mentors were unable to do the final review of the module. My mentors will do with the review before this Friday so I’ll move the module to drupal.org by then.

By the end of the week I’ve created a demo video for the module and fixed some issues. My mentors have already reviewed the code for the work which I did during this week. If anyone is interested, my progress can be followed on my GitHub repo.

Categories:

OSTraining: Would You Be Interested in an Online Drupal 8 Day?

Planet Drupal - Tue, 2016/08/16 - 8:59pm

Drupal 8 has been out since November and is now maturing into a really powerful platform.

We'd like to hold an online event all about Drupal 8. 

The event will be called "Drupal 8 Day" and will be an online version of a Drupal Camp.

It will be one complete day of live presentations. To get a good feel for this type of event, check out Wordsesh.org in the WordPress space.

The goal is to present Drupal 8 in the best possible light, especially to people who normally can't attend a Drupal Camp. We talk with a lot of people who just don't live near any kind of DrupalCamp. Some typical examples are people outside the big Canadian cities, people living in Eastern Europe but outside capital cities, and people in sub-Saharan Africa.

Categories: