This is the second in a series of posts recapping ImageX’s presentations at this year’s DrupalCon.
With so many testing methods available -- code static analysis checks, unit testing, functional testing, front-end performance testing, load testing, visual regression testing, etc. -- it can be difficult for a development team to choose which will work best for their project, particularly with limited time and budget available.
DrupalCon brings together thousands of people throughout the Drupal community who use, design for, develop for, and support the platform. It’s the heartbeat of the Drupal community, where advancements in the platform happen, learnings from its users are shared, and where connections are made that strengthen the community.
As a web agency that specializes in higher education, ImageX keeps its figurative finger on the pulse of the sector. Some weeks are busier than others for new data and studies being released, and this week definitely falls into the busy category. Let’s take a look at the week that was in higher education!
The Bill and Melinda Gates Foundation released a compelling student demographic breakdown of what the higher education landscape looks like in America:
Doris Wong and I sat down at Acquia's Boston HQ to talk about her interesting journey through HTML and frontend work, to UX, to fitness, and finally to Drupal and Acquia, and how even at Acquia it took time, Acquia U and three jobs (!) to really settle in. Here, we talk about that path and what she got out of Acquia U.
Read this blog post on Doris's website for a wonderful, clear introduction to who Doris is and how she got to Acquia and Drupal development.
"Not all [my] experiences were a success, but the one thing I didn’t stop doing was learning. Learning is one of the greatest tools you can have in your arsenal. I first came to Acquia as a UX intern with the goal of finding my next career. When I came to the realization that my heart was in web development, the Acquia U program came into fruition. When the program was finally announced, I hesitated to apply. I didn’t know enough about Drupal to make an educated decision but then I thought, well, why stop learning now?" - Doris WongInterview video - 10 min.
- Name: Doris Wong
- Work affiliation: Digital Marketer, Acquia
- Acquia U profile: To the Front and Back
- Blog/Website: http://dorismwong.com
jam: We are sitting here at Acquia’s headquarters. How many days have you been working here so far?
Doris: I would have to break it down three ways. I have been here for a year with three different departments. I started as a UX Intern--User Experience--with the engineering team. That was for the summer ... No actually, I extended summer. Then from there ...
Backtrack: I was a frontend developer. I teach fitness on the side, and I was kind of looking for a change. Thinking I was going to do fitness, and then I worked at a company for frontend development because I wanted to get back into it who is focused on UX.
Then, I got connected to the UX team here. Then I thought, “Alright, what’s next? I want to try something else”. Then, I met Amy Parker. She kind of convinced me to sign up for Acquia U. I got in. I was at Acquia U, then I was kind of like, “Alright. Let’s see what’s next”. Then I kind of found a role within Marketing, which was not really part of the curriculum, but I developed an interest in them. Right now, I am at Acquia as a Digital Marketer.
jam: Wow. I think we can draw two conclusions from this. The first conclusion we can draw is that you have trouble making up your mind.
jam: The second conclusion we can draw is that three departments in a year at Acquia ... Acquia is a gold mine of opportunities for people who want to try stuff out.
Doris: Definitely. You see that a lot here. You see a lot of people who learn. It is not that I didn’t learn much from the UX or with Acquia U. All of that experience has really helped me with my current role in Marketing, which is focusing on conversion for Acquia.com. So having the User Experience and the Drupal skillset to be able to use certain products on Acquia.com has really helped. In the end, actually surprisingly, all tied in together.
jam: Introduce yourself and tell us something non-Drupal-ly about you.
Doris: Alright. My name is Doris Wong. The non-Drupal-ly thing about me is that I currently teach a dance fitness class called BollyX. It is a Bollywood-inspired dance fitness program. So for those of you who have taken other dance fitness programs, this one introduces the mainstream folks to the world of Bollywood in a fitness format. It’s a lot of fun.
jam: Wow, that’s a cool non-Drupal-ly thing. You had this odd path into Acquia U where you had already touched Acquia - but before you came to Acquia, had you heard of Drupal?
jam: How did you find out about that UX internship that you applied for?
Doris: This is where the power of networking comes in. At my internship ... It was an internship doing frontend development. It was more email marketing, but the company was involved with UX consulting or research. The former director of UX at Acquia, she had previously worked at that company. That company was probably the size of maybe 10 folks, so it is definitely a small company, very personal, intimate. I developed a relationship with the CEO. The CEO and the former director of UX here were good friends. Upon my exit, I expressed interest in developing a career in UX. During that time, the director was looking for an intern and hence the connection there. Definitely network.Meet Acquia U
jam: What were your expectations going into Acquia U?
Doris: That’s an interesting question. I don’t think I had a lot of expectations. I feel that in my experience, I didn’t have any expectations going into anything new, because it is a new program. I wanted to have a clear head going in, as to not have it affect my experience in the program. I was going in knowing that this was kind of the rebirth of the program since there was kind of a gap between the first program and this. So I knew it was going to be kind of like, so to speak, a pilot. There are probably going to be some hiccups and I knew that I would kind of have to go with it and run with it. Just having an open mind really helped me.
jam: What did you want to get out of it?
Doris: A sense of ... let me think about that. I think just kind of getting a better understanding of where I am in terms of my career and skills. It kind of helped me understand who I was and what I am capable of. Because Drupal was new, I was kind of coming from scratch. Didn’t know anything. It really helped me figure out, “Alright, this is what I can do. This is what I am capable of. What’s next?”
jam: Alright. How was it?
Doris: It was challenging. It was really challenging because it was a little more fast-paced. There were some folks within the program that had Drupal experience, some who have built sites around. I think for me having a frontend development experience really helps. Knowing a little bit – at least a general understanding of PHP, I knew the frontend stuff like CSS, HTML kind of helped, but I think it was more of the site building component and just understanding how it worked together. That was the more challenging aspect of it. But that was just the beginning. As I keep getting engrossed to it, it just got a lot easier.
jam: Run us through a day at Acquia U.
Doris: There is no typical day. But I can say that as you are going through the program, you would come in – usually the first half would be lecture. Going over ... reviews. That is also dependent on where we are. If we feel comfortable passing a subject, if we are familiar, we can move on to the next. Then, the second half of the day could be more of a project that we are working on. If we wanted to do more learning, there was some flexibility in the schedule. That was the great thing about the program, there is a lot of flexibility during the day. We will have a set schedule, but depending on what we wanted to do, the instructor was really great in allowing us to decide, “I think we want to focus on this today versus that”.
jam: What’s the most important thing you got out of going through Acquia U?
Doris: Definitely working with the team and learning more about the Acquia culture. That’s the one thing that is really good about this program is because you are not just sitting in a random office or something that you are just learning something. You are actually engrossed in the culture, so you get an understanding for what it is like working here and working with the people around you.
jam: What’s the one piece of advice that you would give for people looking for a new career?
Doris: One piece of advice is that--this may sound interesting--but you need to be dedicated. It is not easy making the transition. If you are serious about making a career change, then you want to be devoted to it and do everything you can to get better at it because you can just go in and say, “Maybe I will just try this and not do anything about it”. You are not getting much value. You have to have dedication.
jam: Would you recommend Acquia U to others?
Doris: Yes. Not only are you learning a lot from the instructors, you are also learning from your peers because we have folks that are coming from different careers and I feel that anything that you learn, you can always translate that in whatever career you end up choosing afterwards.
jam: Awesome. Thank you!
Doris: Thank you! It’s a pleasure.Skill Level: Beginner
TGIF! We hope you're having a great week and are gearing up for an even better weekend. Thanks for joining us for Episode 10 of The Mediacurrent Friday 5!
Palantir: On the Air with Palantir podcast, Ep. 05: Consulting engagements that work - a case study with Rhodes College
Welcome to a new episode of On the Air with Palantir, a long-form podcast by palantir.net where we go in-depth on topics related to the business of web design and development. It’s June 2016 and this is episode #5. In this episode Account Manager Allison Manley is joined by our client Justin McGregor from Rhodes College. Allison caught up with Justin at DrupalCon in New Orleans last month, and spoke with him about how his school has implemented Drupal, how we worked together, and how it’s been going since.iTunes | RSS Feed | Download
We'll be back next Tuesday with another episode of the Secret Sauce and a new installment of our long-form interview podcast On the Air With Palantir next month, but for now subscribe to all of our episodes over on iTunes.Need a helping hand with effective consulting? Our years of expertise can help make sense of any challenge you're facing with your web project, whether strategy, design, development, or ongoing support in nature.
Look for our transcript of this episode added here soon.
Tips and tricks for working with REST module in Drupal 8.
The Public Liberties and Human Rights Center was founded in 2008 as a "desk" within Al Jazeera. Today it is staffed by a diverse team which works across different areas of the network. The editorial team operates on all of Al Jazeera's platforms and in all of the network's languages. The journalists produce original stories and content examining human rights issues around the world, complementing the network's hard-hitting news. Their website contains information about initiatives and events organized by the Center as well as many images and videos related to their projects. In 2015, Vardot developed a new website to bring together all the work Al Jazeera does regarding human right through all its platforms to one hub.
Goal of the project
The goal of Vardot was to build an editor-friendly distribution that will also bring visitors a seamless user experience. The mission was to launch a modern multilingual SEO-optimized website that will be integrated with social networks and have the ability to handle a high traffic.
The right CMS for media networks
Vardot already partnered with Al Jazeera to develop other AJ websites such as Sharek, Forum, Stream and Cafe. All these sites were built on Drupal because the client was looking for a CMS that will be able to handle high traffic, different permission levels, a large number of subpages and at the same time be secure and flexible for users. For the AJ Public Liberties project, we used our very own Drupal 7 distribution, Uber Publisher, that in our opinion ideally meets the needs of the Media producers and Online News Publishers.
The idea of this distribution is easy: most of news websites require the same package of features such as the ability to upload and edit text quickly, create roles and permissions for a better security, add taxonomy terms to organize the content in the convenient way, handle high traffic and more. Uber Publisher does just that. It is a combination of modules, configurations, settings and custom development for online publishers.
Uber Publisher as a competitive advantage
Having the “foundation” of the website ready, our team was able to concentrate on the uniqueness of the Liberties website without wasting time for a basic coding. As a result, the work on the new website of Al Jazeera Public Liberties & Human Rights Centre took less than one month.
Furthermore, we were able to to provide Al Jazeera with many different ways to change the display of content listings to meet their needs at any time. Depending on what Al Jazeera content managers wanted to showcase, the site allows them to feature one main article or multiple articles to tell a story and promote the most important news first.
Building a good website in just one month is very impressive, but we are looking forward to be able to work even faster. After updating Uber Publisher and making it more universal for any kind of online publishing company we expect to decrease not only the time of the development, but also the average total cost of ownership of news websites.
Building sites for enterprise companies always means accepting a big challenge, but our team likes to exceed expectations. Working with Al Jazeera before we’ve learned a lot about our client’s assumptions and ways of working, and this time we’ve just used the experience collected before and made a high-quality website in less than one month. Like Al Jazeera Public Liberties & Human Rights Centre contributes a lot to the awareness of humanitarian organizations, Vardot is happy to add some improvements to Drupal and share our experience with you.Tags: Drupal Planet Title: Rights and Liberty on Your Screen: New Website of Al Jazeera Media Network
We developed the WalkHub Help Widget to help site owners guide their visitors to the right documentation content - right inside an application or web site. You can also set up your own widget creation interface easily - in this blogpost, we will guide you through the steps.
One of the major features of Drupal 8 is the Configuration Management Initiative (CMI). I’ll briefly touch on why CMI is so important, but our own Alex Pott has written extensively on the creation, principles and importance of this feature.
Previous to Drupal 8, it was cumbersome to trade configuration between sites and environments. Moving configuration changes from a developmental environment into production meant either making the changes manually through the UI, or utilizing an advanced option like Features. A third option would be full-database swapping, but that can be quite risky and is generally considered poor-form.
The goal of automated testing is confidence: confidence in application stability, and confidence that new features work as intended. Continuous integration as a philosophy is about speeding the rate of change while keeping stability. As the number of contributing programmers increase, the need to have automated testing as a means to prove stability increases.
This post is focused on how the automated testing infrastructure on Drupal.org works, not actually writing tests. Much more detail about how to write tests during Drupal development can be found in community documentation:
DrupalCI essentially runs two categories of tests:
Functional tests (also called blackbox testing) are the most common type of test run on DrupalCI hardware. These tests run assertions that test functionality by installing Drupal with a fresh database and then exercising that installation by inserting data and confirming the assertions complete. Front-end tests and behavior driven tests (BDD) tend to be functional. Upgrade tests are a type of functional tests that run a full installation of Drupal, then run upgrade commands.
Unit tests run assertions that test a unit of code and do not require a database installation. This means they execute very quickly. Because of its architecture, Drupal 8 has much more unit test coverage than Drupal 7.
These test categories can be broken down further into more specific test types.What testing means at the scale of Drupal
Drupal 8, with its 3,000+ core contributors and 7,288 contrib developers (so far), needs testing as a means to comfortably move forward code that everyone can trust to be stable.
Between January and May 2016, 90,364 test runs were triggered in DrupalCI. That is about 18,000 test runs requested per month. Maintainers set whether they want tests to run on demand, with every patch submitted, or nightly. They also determine what environments those tests will run on; there are 6 combinations of PHP and database engines available for maintainers to choose from.
The majority of these test runs are Drupal 8 tests at this point. (19,599 core tests and 47,713 contrib project tests were run during those 5 months.) Each test costs about 12 cents to run on Amazon Web Services. At the time of writing this post, we averaged around $2,000 per month in testing costs for our community. (Thank you supporters!)An overly simple history of automated testing for Drupal
Automated testing first became a thing for Drupal contributed projects during Drupal version 4.5 with the introduction of the SimpleTest module. It was not until Drupal 6 that we started manually building out testbots and running these tests on Drupal.org hardware.
In Drupal 7, SimpleTest was brought into Drupal Core. (More information about what that took can be reviewed in the SimpleTest Roadmap for Drupal 7.)
In Drupal 8, PHPUnit testing was added to Drupal Core. PHPUnit tests are much faster than a full functional test in SimpleTest—though runtest.sh still triggers a combination of these test types in Drupal 8.
The actual implementation of automated testing was much more complicated than this history suggests. The original testbot infrastructure that ran for 7 years on Drupal.org hardware was manually managed by some fiercely dedicated volunteers. The manual nature of that maintenance led to the architecture of DrupalCI, which was meant to make it easier to test locally at first and later focused on autoscaling on powerful hardware that could plow through tests more quickly.DrupalCI's basic structure
In The Drupal.org Complexity, we could see the intricate ways that Drupal's code base interacts with other parts of the system.
We could further break out how systems like DrupalCI are interrelated.
DrupalCI is a combination of data stored on Drupal.org, cron jobs, drush commands, and most importantly a couple of Jenkins installations to manage all the automation.
Jenkins is the open source automation server project that makes most of the system possible. We use it for automating our build process and deploying code to our dev, staging and production environments. It automates just about anything and is used by companies small and large to run continuous integration or continuous deployment for their applications. It's considered a "best practice" solution alongside options like Travis, CircleCI, and Bamboo. They all have slightly different features, but automation is at the core of most of these DevOps tools.
To provide continuously integrated tests, you need to trigger those tests at a moment when the tests will have the greatest value.
The three triggers for running a test job are when a patch is added to an issue comment, when code is committed to a repository or daily on a cron. Maintainers can specify which triggers are associated with which branches of their projects and which environments should run those tests.
For core these settings look something like this:
This detail allows for specific tests to run at specific times per the Drupal.org Testing Policy for DrupalCI.
To make this automation happen, we have an installation of Jenkins (Infrastructure Jenkins below) that is polled by Drupal.org once per minute with testing jobs to be queued.
These jobs live in a database record alongside Drupal.org.
Infrastructure Jenkins speaks to the CI Dispatcher (also Jenkins) where the testing queue regularly passes those jobs to available testbots. CI Dispatcher uses an Amazon Web Services EC2 plugin to spin up new testbots when no existing testbot is available. Each testbot can spin up Docker containers with the specific test images defined by the maintainer. Theses containers pull from DockerHub repositories with official combinations of PHP and database engines that Drupal supports.
After a testbot is running, the CI Dispatcher is in constant communication with the bots. You can even click through to the console on CI Dispatcher and watch the tests run. (It's not very exciting—perhaps we should add sound effects to the failures—but it is very handy.)
Once per minute, Drupal.org polls the CI Dispatcher for test status. It responds with pending, running, failed or passed. Failed and passed tests are then pulled back into Drupal.org for display using the Jenkins JSON API.
Tests can also be run on demand at the patch, commit or branch level using the handy "add test" and "retest" links.Why did we build this ourselves? Why not use [insert testing platform here]
Lot's of people have asked why we don't use TravisCI, CircleCI or some other hosted testing solution. The short answer is that most publicly available testing systems require Github authentication and integration.
Additionally, our testing infrastructure is powerful because of its integration with our issue queues. Read the aforementioned The Drupal.org Complexity for more information.
Another reason to run our own testing is scale. To get through all of the core tests for Drupal 8 in an acceptable amount of time (about 44 mins on average), we run very large testbots. These bots have 2 processors with 8 hardware cores. With hyperthreading, that means we have 32 hardware execution threads—about 88 EC2 compute units. They are not exactly super computers, but they are very performant.
We average nearly 18,000 test runs per month. During our peak usage we spin up as many as 25 testbot machines—though usually we cap at 15 bots to keep costs under control. This helps us plow through our testing needs during sprints at DrupalCons and large camps.
We have explored using an enterprise licensed version of either Github or CircleCI with our own hardware to tackle testing. That same consideration has been given to SauceLabs for front-end testing. Right now, there is not a cost savings to tackle this migration, but that does not rule it out in the future. Testing services continues to evolve.Accelerating Drupal 8
In my first months as CTO, I was told repeatedly that the most important thing for us to work on was testing for Drupal 8. In those early days as I built out the team, we were mostly focused on catching up from the Drupal 7 upgrade and tackling technical debt issues that cropped up. In DrupalCon Austin, I had members of my team learn how to maintain the testbot infrastructure so that we could take over the process of spinning up bots and dealing with spikes in demand.
By early 2015, we had optimized the old testbots as much as they were going to be optimized. We moved them to AWS so we could spin up faster machines and more bots, but there were features that were waiting on the new DrupalCI infrastructure that were blocking key development on Drupal 8.
In March of 2015, we invited all the community developers that had helped with DrupalCI to the Drupal Association offices in Portland and sprinted with them to figure out the remaining implementation needs. The next couple of months involved tweaking DrupalCI's architecture and cutting out any nice to have features to get something launched as soon as possible.
It is no coincidence that from the time of DrupalCI's launch until the release of Drupal 8, progress was rapidly accelerated.
I am immensely proud of the work of all the community members and staff that worked directly with core maintainers to unblock Drupal 8 development and make it faster. This work was critical.
Thank you to jthorson, ricardoamaro, nick_schuch, dasrecht, basic, isntall, drumm, mikey_p, mixologic, hestenet, chx, mile23, alexpott, dawehner, Shyamala, and webchick. You all made DrupalCI. (And huge apologies to all those I'm undoubtedly leaving out.) Also thank you to anyone who chimed in on IRC or in the issue queues to help us track down bugs and improve the service.What's next for testing Drupal
Most of the future state of testing is outlined in the Drupal.org Testing Policy for DrupalCI.
Key issues that we still need to solve are related to concurrent testing improvements and new test types to support. While we have PhantomJS integrated with the test runner, there are optimizations that need to happen.
Testing is not an endpoint. Like much of our work, it is an ongoing effort to continuously improve Drupal by providing a tool that improves how we test, what we test, and when we test.
You want to develop with Drupal 8? We show you some useful sources of lectures, examples and documentations that will ease your work.
Hi geeks! Did the post title get you excited? Great, because Organic Groups (OG) and Message stack are getting closer to being Drupal 8 ready.
I’d like to give an overview about their state, the amazing community effort around them, and also share some of my personal thoughts about contribution to Drupal 8 in general.Organic Groups
For years Organic Groups has been one of the proven solutions for multi-sites functionality, in the form of one code base, one database, and one dashboard to rule them all. After so many years and seeing so many different implementations, such as Harvard’s OpenScholar, OpenAtrium, and many others, I’m even more confident OG is doing many things right.
Most of OG7’s concepts are being migrated to OG8, but obviously this is also a good time to fix some old mistakes. One of the mistakes was treating users and content (i.e. non-user entities) alike. But, well, you know - they are not. Because when we came to re-think of it, membership really makes sense only for users. For example, if the membership state is active, pending or blocked, that should indeed be applied only to users. So we’ve changed it: