A versatile solution for importing large CSV files into Drupal.In this post we will cover...
- Point one
- Point two
- Point three
Migrating content from an existing site or an external data source can help reduce the effort required by content editors to get a new site ready for launch. As a result, constructing and executing content migrations is a common task we undertake as part of the site build process. While these migrations can vary in type, typically spreadsheets are exported in a comma separated value (CSV) format due to their simplicity.
While Drupal has robust support for migrating in from a CSV file, the current structure can struggle when presented with large CSV files. In particular, the migration import process can run out of memory part way through the migration process. We encountered this problem while migrating tens of thousands of locations for a client. Increasing the PHP memory limit for the migration was an initial step, but proved not to be enough:
- Memory usage is 1.21 GB (80% of limit 1.51 GB), reclaiming memory.
Memory usage is now 1.21 GB (80% of limit 1.51 GB), not enough reclaimed, starting new batch
Even though the migration module attempts to reclaim memory and start a new batch, the process does not always complete.
Some approaches to get around this issue include scripting your migration and utilizing the limit option when running a migration. However, we wanted a solution that could be more versatile and wouldn’t require custom scripting for each new migration we would write.
As a result, we wrote a custom Drush command that acts as a wrapper around the default Migrate import command. Our custom command splits a large CSV file into smaller files that can be imported in batches.
As an example, the following command may be run:
drush migrate:import:batch sample_migration --batch-size=100
When the migration is run, the CSV source file for the sample_migration is split into smaller CSV files with 100 lines each. The migration runs for each of these files. These files are temporarily stored in the private files directory and are cleaned up after the migration is finished.
Other migration operations run like normal and all of the default options may be passed in. Migration mapping hashes are maintained, so the migration may be rolled back like normal, too.
The module’s code currently exists in a Github repository which also contains more information on the module’s usage, but we plan on releasing it as a contributed module on Drupal.org in the future. Feel free to give it a try on your project and let us know how it works for you!Development Drupal Site Building
Having more companies working with Drupal is a good and necessary thing, but it means we need to improve the way that we onboard, recognize, and differentiate those who help sustain and innovate Drupal.
A few weeks ago, I earned my first ever Drupal contribution credit for my DrupalCamp Colorado keynote. While I am oddly excited about that, I also find it somewhat ironic, as that keynote should not be mistaken for my first contribution to Drupal.
According to my Drupal.org profile, I’ve been a community member for over twelve years. In that time, I’ve presented keynotes for three other DrupalCamps, presented sessions and participated in panels going back to DrupalCon Boston 2008, led the RFP process for the redesign of Drupal.org, chaired DrupalCon Chicago 2011, served on the board of the Drupal Association for nine years and, most recently, served on the Executive Director Search committee. That is but a partial tally of my individual contributions; of course my company, Palantir.net, has also made considerable contributions of time, talent, and treasure over all these years.
Recognition is not my motivation for these efforts; like so many open source contributors, I give back to Drupal because I am committed to stepping up when I see a need or an opportunity. When I was new to the community, the karma earned from such efforts, code and non-code, was informally held in the living memory of those who were there. I always felt that I had earned the credibility and support of those with whom I collaborated closely to move on to the next opportunity, to tackle and solve the next problem. In many ways, as a woman on/of the internet, I appreciated the relative anonymity of it.
In that way, Drupal has become the largest independent community-driven open source project. And many of us believed that our collective success and the impact we made was enough to sustain the virtuous cycle of open source. But was it?
Open source has won: we now have legions of people and companies who rely on Drupal and other open source tools and products; however, these companies picked the best tool, which just happened to be an open source tool, and they don’t necessarily yet know the open source way. Twelve years ago, the Drupal community was small enough that those established norms and expectations were passed on person-to-person, along with the lore and the legends. The old ways of influencing behavior and enforcing norms through social bonds (aka peer pressure) aren’t strong or explicit enough for the swells of newcomers.
There is a lack of shared understanding, visibility, and support for what it takes to not just keep Drupal sustainable, but to have it thrive and win in a competitive landscape. This lack of clarity has led to the emergence of multiple subcultures within the commercial ecosystem and a worrying disparity between those who benefit the most from Drupal versus those who give the most.
In his Amsterdam 2014 keynote, Dries noted that while open source has a long history of credit (for code) to the individual contributors, this does not adequately recognize (or incentivize) the organizations. He proposed a simple way to give organizations credit in addition to individual credits for the core issues their teams either performed directly or sponsored, which the Drupal Association released in late 2015. Over time, this system has been expanded to capture more than just code contributions.
And yet, the contribution credit system has not wholly replaced karma. As my own experience shows, so much of the vital work that Drupal relies on is not yet captured in credits. Due to my privilege (not looking for a job, having well-established connections in the community, etc.), the lack of visibility was a feature, not a bug, for me as an individual contributor.
However, wearing my Palantir CEO hat I’ve come to realize that the failure to capture fully what and how companies do (and are expected to) contribute is far more problematic for the sustainability of the project. Some of the most essential work in the community (Drupal Association Board of Directors, the Community Working Group (CWG), the Security Team and non-code Core team work including release management, communication, sprint organizing, and overall project and initiative coordination) is severely undervalued or all-in-all ignored by the contribution system. George DeMet's ongoing commitments as the chair of the CWG often average anywhere from ¼ - ½ of his time (more at intense times) and over the last year he received four credits (the other members of the CWG received even less!). The community and the project suffer because this invisibility obscures, and indeed over time deteriorates, the community expectations and norms by measuring what is easy to measure, rather than what matters.
When Drupal 7 was released, the firms that built Drupal enjoyed a competitive advantage: those who wanted to use Drupal knew which firms meaningfully contributed and why it mattered. However, over the last five years, the Drupal ecosystem has expanded to include many new, larger firms that leveraged partnership and sponsorship programs to establish their Drupal credentials.
These programs and the new implementers and agencies they ushered into the Drupal community are essential to Drupal’s growth and adoption. They are a welcome addition to the ecosystem. However, there are serious problems with the ways that these programs have been structured to date and their unintended impact on our culture of contribution:
- Status within these programs is primarily pay-to-play and non-financial contributions to the project are not required.
- The programs do not directly support or indirectly incentivize the time or talent contributions on which the Drupal project depends.
- The financial proceeds of such programs benefit infrastructure initiatives (Drupal.org and more broadly the Association) and market visibility, which are not necessarily the areas of greatest need for the project or community.
- These programs have undermined the reputational system that prioritized successful outcomes (successful client implementations AND contributions back to the project) and replaced it with one that favored outputs (financial success and client list).
Allowing companies to position themselves as leading experts in Drupal without validation that these firms are contributing commensurate with the benefits derived from Drupal has been corrosive to the sustainability of the project. This has tacitly supported the commoditization of Drupal services, devalued the competitive advantage received from direct contribution, and simultaneously incentivized and conditioned all in the ecosystem to increase indirect contribution (sponsorship and advertising on Drupal.org and events including DrupalCon).
As I noted on a panel at OSCON, I see all of this as a success problem. Having more companies, including large scale implementers and agencies, working with Drupal is a good and necessary thing. What we need to improve is the way that we onboard, recognize, and differentiate those who help sustain and innovate Drupal to (re)establish a culture of contribution for Drupal. Doing this well will involve creating new and easy-to-access avenues for contribution that match the project’s weighted needs and companies’ available resources (be they time, talent or treasure). A concerted focus on what matters will shore up Drupal’s path to long-term sustainability.Community Drupal Open Source People
Happy New Year, everyone! If you’re not willing to let go of the holiday spirit just yet, you’re in luck - we’ve prepared an overview of our favorite Drupal blog posts from December. We hope you enjoy revisiting them!READ MORE
Mike and Matt talk with the leads of Drupal's "Automatic Updates" initiative to discuss the security, limitations, status, and gotchas of automatic updates.
We're thrilled to announce that we will be back in Berkeley on October 14th - 17th, 2020!
We are stoked to be returning to The Hotel Shattuck for 4 days of roaring sessions, training classes, summits, sponsors, and parties for all you flappers, philosophers, and Drupallers!
We’re still working on all the details, so if you want to follow along with news hot off the presses, sign up for our mailing list!
As 2019 comes to an end and we wrap up another decade of Drupalling, we’re excited to announce another BADCamp that’s sure to be the bee’s knees, and invite you to join us over at 2020.badcamp.org. We'll be back in Berkeley on October 14th - 17th, 2020!
We are stoked to be returning to The Hotel Shattuck for 4 days of roaring sessions, training classes, summits, sponsors, and parties for all you flappers, philosophers, and Drupallers!Drupal Planet
At DrupalCon Amsterdam we announced the formation of a Contribution Recognition Committee. The purpose of this committee is to recommend solutions for how we recognize contributions to the Drupal project made by both individual and organizational contributors, and to advise the Drupal Association on how to weight each type of contribution relative to the others.
We kicked off the process with this call to action, seeking volunteers to participate on the committee who represented a variety of personas within the Drupal community:
We saw a tremendous response to this call to action, and the committee has now been assembled. The members are:
Committee Chair: Mike Lamb
Mike Lamb has just completed his second term on the Drupal Association board. He has been a member of the Drupal community for 8 years and is the Vice President of Global Digital Platforms at Pfizer. He brings his perspective as a former board member, a leader of digital teams, and a representative of a large end-user of Drupal to the table. His particular interest is in finding ways to incentivize other end-user organizations to make contribution history part of their vendor selection process.
Committee Member: Tiffany Farriss
Tiffany Farriss has a long history of leadership and service in the Drupal community, and is the CEO and co-owner of Palantir.net. She served on the Drupal Association board for 9 years, most of those as Treasurer; chaired the organizing committee for DrupalCon Chicago in 2011; and has served in many other ways, most recently on the executive search committee for the new Drupal Association Executive Director. Tiffany has personal experience across many kinds of contribution, as well as leading Palantir to be a top contributing organization.
Committee Member: Ryan Szrama
Ryan Szrama is well known for his work on Drupal Commerce and for co-founding Centarro (formerly Commerce Guys) in 2009. He's been a member of the Drupal community for nearly 14 years, was elected by the community to serve on the Drupal Association Board of Directors in 2017, and has just been appointed to another 3-year term. He lends his perspective as a Drupal business leader and D.A. board member to committee discussions and is interested to see how the data we gather about contributions can be used in ways beyond just crediting contributors.
Committee Member: Stella Power
Stella Power is a respected leader in the Drupal community in Ireland and is the Managing Director of Annertech. In her almost 14-year journey with Drupal, she's done everything from contributing code to volunteering at and organizing local events, consistently serving as a volunteer organizer of DrupalCon and Trivia Night quiz master. She provides the committee with her extensive experience in all of these areas - and as someone with connections to contributors around the world.
Committee Member: Jacob Rocowitz
Jacob Rockowitz maintains the Webform module for Drupal 8, as an independent Drupal consultant. Jacob has extensive experience collaborating with dozens if not hundreds of developers because of the wide usage of Webform, and he has pioneered a number of efforts to improve the sustainability of open source. He brings a unique perspective as a sole proprietor, as well as the knowledge gained from these sustainability experiments. One aspect of this project that is of particular importance to Jake is ensuring that the system incentivizes the quality of contributions, not just their quantity.
Committee Member: Rakhi Mandhania
Rakhi Mandhania is a Project Manager Colab Cooperative and co-founder of the DrupalCampus Ambassador Program. For more than six years in the Drupal community Rakhi has been a frequent attendee at local events in India, as well as international events like DrupalCon in Asia, North America, and Europe. Rakhi brings the perspective of Drupal users in the higher education space, as well as her experience as a project manager.
Committee Member: Paul Johnson
Paul Johnson has been working with Drupal since 2006 and is the Drupal Director at CTI Digital. Much of his client work is focused on the nonprofit world, and he brings that experience as well as knowledge of how to encourage for-profit, closed organizations to become more open. Among his many contributions to the project, Paul has focused on international social media efforts, including promotion from the @Drupal Twitter handle, and initiatives like #CelebrateDrupal8. His more recent experience includes work on the Promote Drupal initiative, and specifically the Drupal pitch deck. His passion lies in helping the Drupal project to realise the potential of its scale, ensuring contributions of all forms and scale are recognised and valued in particular beyond code.
These committee members are by no means the only community members who will be involved in the process. Six people are not enough to represent the breadth of diversity in our community. This is why the committee, both collectively and individually, will be seeking out feedback from every corner of our community to feed this process.
They'll also be working closely with the Drupal Association itself, particularly myself (Tim Lehnen) as CTO and maintainer of the system, and Rachel Lawson as our community liaison who already has tremendous insight from the community feedback she's gathered on this subject to date.
The committee will be meeting regularly over the next several months, with the goal of reporting significant progress and first recommendations at DrupalCon Minneapolis in May of 2020.
groups.drupal.org frontpage posts: Drupal 9 meetings now every Monday at 7pm UTC, help us prepare for beta!
We started the "All things Drupal 9" meetings all the way back in October 2018 to start preparing for the new major release. Now in 2020, the year of Drupal 9, it was natural to make the meeting happen every Monday given the target dates coming up fast.
Contributed project maintainers, site owners, core developers are all welcome. If you are planning a Drupal 9 project for later in the year, this is your place as well. We open the floor with equal opportunity for people to propose topics, so we can cover your pressing questions as well! Join at 7pm UTC in the #d9readiness channel on Drupal Slack any Monday. The next meeting will be on January 13th, 2020.
We've been promoting Drupal 9's June 3rd 2020 target release date a lot, but that can only happen if the Drupal 9.0.0 beta requirements are all done by the end of February 2020, which is coming up real fast. There are still a lot to do and we need your help! If that does not happen, then the release will happen in August or December. While that would give less time for Drupal 8 users to update, we cannot compromise on the stability of Drupal 9 out of the gate.
Check out the detailed alternate timelines at https://www.drupal.org/core/release-cycle-overview
Ivan Stegic and Tess Flynn discuss migrating Drupal 7 sites to Drupal 8.Guests
Tess Flynn, TEN7 DevOps, and Ivan Stegic, TEN7 Founder and PresidentTranscript
Tess: You're listening to the TEN7.com audiocast. We're here to discuss Drupal, migration, technology, and I'm here with -
Ivan: Ivan Stegic
Jonathan: Jonathan Freed
Last year was a very productive for Drupal Commerce. We saw continued growth in overall usage, community contribution, and core feature availability. We’ve got big plans for 2020 but first wanted to pause to reflect on what we accomplished in 2019.A year of growth
We started last year with 7,000 reported installs. Since then, we’ve gained another 3,000 - a 40% increase! The number of community members on Drupal Slack (in the #commerce channel) grew from 1,000 to 1,560. The number of listed payment gateways, a metric we use to track expansion to markets around the globe, grew from 90 to 138 with more on the way.Read more
Over the last two years, we have managed to make some significant improvements to the DrupalCon Code of Conduct (CoC) so that it:
- Is more clear what is and is not acceptable behavior
- Details ways to raise issues through to the CoC Team for action
- Lists exactly who on the CoC Team handles issues
- Requires leaders — such as speakers, sponsors, etc. — to be held to the highest standards of all
- Explains the consequences of unacceptable behavior.
Thank you to everyone who has been involved, both staff and community members, for their work.
We have also taken the opportunity to train members of Drupal Association staff in CoC management through the course provided by Sage Sharp at OtterTech, organised by the Drupal Community Working Group. If you organise events, we highly recommend attending this online course in 2020.
No Code of Conduct can ever be considered “final”, however, and the Drupal Association have a process of making a release of a new DrupalCon Code of Conduct in the first quarter of each year, including any approved updates collected during they previous year.
It is that time again to call upon the Drupal Community to propose any refinements they see beneficial, by creating issues in the project issue queue. Anyone with a Drupal.org user account can create an issue and you do not need technical expertise to do so.
You will notice that I have already raised one issue, related to adding caste as a protected characteristic alongside others such as gender and race.
I look forward to reviewing your proposed refinements and continuing to build and maintain a world class Code of Conduct for our world class tech conference.
2019 comes to a close, and with it, another year of fascinating podcast guests that taught and inspired us. We’ve selected some of our favorite clips from this year’s guests for you to enjoy. Listen to the full podcasts if you like what you hear.
Happy New Year to all! 2019 was definitely a year we won’t soon forget, as it marked some of Agiledrop’s biggest successes to date: a significant expansion of our team, offices and technologies, topped with an even greater focus on providing the best possible experience for our employees. We are now more than ready to take on not only this new year, but rather the whole new decade.READ MORE
On 7 January, 2020, the Drupal module JSON:API 1.x was officially marked unsupported. This date was chosen because it is exactly 1 year after the release of JSON:API 2.0, the version of JSON:API that was eventually committed to core. Since then, the JSON:API maintainers have been urging users to upgrade to the 2.x branch and then to switch to the Drupal core version.
We understand that there are still users remaining on the 1.x branch. We will maintain security coverage of the 8.x-1.x branch for 90 days. That is, on 6 April, 2020, all support for JSON:API, not in Drupal core, will end. Please upgrade your sites accordingly.
Happy New Year!!! Our normally scheduled call to chat about all things Drupal and nonprofits will happen Thursday, January 16, at 1pm ET / 10am PT. (Convert to your local time zone.)
This month, in addition to our usual free-for-all, we'll be talking about Drupal and CiviCRM. Have you got it up and working your Drupal 8 site? For those of us still working in Drupal 7, what can or should we doing to prepare for the inevitable upgrade? What are your favorite resources for working with these two systems? Come share your experiences!
Feel free to share your thoughts and discussion points ahead of time in our collaborative Google doc: https://nten.org/drupal/notes
We have an hour to chat so bring your best Drupal topics and let's do this thing!
This free call is sponsored by NTEN.org but open to everyone.
REMINDER: New call-in information -- we're on Zoom now!
- Join the call: https://zoom.us/j/308614035
- Meeting ID: 308 614 035
- One tap mobile
- +16699006833,,308614035# US (San Jose)
- +16465588656,,308614035# US (New York)
- Dial by your location
- +1 669 900 6833 US (San Jose)
- +1 646 558 8656 US (New York)
- Follow along on Google Docs: https://nten.org/drupal/notes
- Follow along on Twitter: #npdrupal
This blog post shows you how to access your lando databases using tools like PHPStorm, Sequel PRO and Command Line.
You’ve just freshly installed Drupal 8 on your computer and you are really looking forward to
getting your website up and running fast. You want all those amazing features you have heard
about of Drupal 8. So, what do you do next? Pick your Drupal 8 modules! Coz modules are the
building blocks of any great Drupal mansion.
Of course, it isn’t as simple as it sounds. Drupal 8 is powered by its ever-growing open-source
community who work hard to develop thousands of modules that can enhance and extend the
functionality of a Drupal website. Choosing out of thousands of Drupal 8 modules can be a
Admin Toolbar Module
Luckily for you, many significant contributed modules from Drupal 7 have made their way to
Drupal 8 core. Which means, you will only need to enable these modules once Drupal 8 is
installed. Let’s dive into a short list of top Drupal 8 modules you must have to build great digital
experiences. Following that, we’ll talk about how to pick your Drupal 8 modules.
The Drupal Admin Toolbar module is extremely useful for better navigation for site admins and
site builders. The default Toolbar isn’t very user friendly. This module aims to extend the
functionality of the toolbar by offering drop-down menus that allow easy access to various
admin pages. It also offers a submodule to extend its functionality called Admin Toolbar Extra
Tools. This submodule adds more helpful links to the admin menu to perform tasks like flush
cache, run cron and more.
The Drupal 8 Paragraphs Module gives a lot of power in the hands of content authors and
editors. It allows them to create flexible and structured content easily. You can now add various
paragraphs field types like images, text blocks, quotes, slideshows, videos and so much more.
Using the familiar node edit form, you can add/play around with as many paragraph items and
place them wherever you want to. It also offers widgets (still in experimental) which will allow drag and drop functionality, duplicating paragraphs and many other features that can elevate
user experience while working with the Paragraphs module.
This is one Drupal 8 module that is always on my recommended-modules-list. It is so versatile
and such an essential Drupal module that it is hard to ignore. Almost every website needs a
form. A contact form or a survey form or a feedback form – there is no running away from
them. The Drupal 8 Webform Module is a rich form builder that is easily customizable and
extendable. It allows to collect form data, send it to third-party applications or send emails to
admins or users. You can also export this data to spreadsheets for further analysis. There’s so
much more that the Webform module offers which cannot be summed up here.
The Drupal Display Suite module is a very easy-to-use and handy tool to build page layouts. It
offers a slick drag and drop interface to arrange content. There are a list of predefined layouts
and templates to choose from. Custom layouts and templates can also be created and added in
the theme. Custom view modes and custom fields can also be defined.
The Drupal Devel module is a very handy tool for developers and site admins. It is widely used
in testing purposes because of its ability to generate a lot of content for nodes, comments,
users and various content types and entities. Also, it allows developers debug any problems
with node access. Page footers can be added for all pages with the help of its submodule called
Webprofiler. Webprofiler also gives the site admins an insight to some analytics about the
caching abilities, database queries, resource utilization and much more.
GraphQL is a modern querying language that replaces old-timers like REST to communicate with
APIs. It is faster and yields only the results you are looking for – minus all the unwanted
baggage that comes along with a REST API call. The Drupal GraphQL module enables your
Drupal website to create GraphQL schemas and expose Drupal entities with GraphQL client
Having a well-structured URL does not only improve the user experience, it is very vital for
search engine optimization too. The Drupal Pathauto module is definitely a must-have module
in every Drupal project. It helps in automatically generating SEO friendly and well-structured
URLs. Site admins can also change the pattern system by changing the tokens it uses.
The Drupal Google Analytics module adds Google analytics tracking system to your website.
Using this Drupal module, all features of Google analytics can be accessed and integrated with
your website. It allows for domain tracking, users tracking, monitoring tracked links, monitoring
downloaded files, site search, Adsense support and much more.
Be it personalization modules, social media integration modules, marketing automation
modules or any other module, you can always find more than one for each functionality. Here’s
what you should keep in mind before making a choice:
Before you download a module, you should know if it is going to be compatible with your
version of Drupal. You just cannot install a version 7 module into your Drupal 8 installation
without cross checking if it is supported. To find out what version of the module has been
released in Drupal.org, you can go to the module’s project page and scroll right to the end
where you will see the download link and find the release versions specified. If it isn’t specified,
go to http://drupal.org/project/Modules/name and filter by Core compatibility.
You must keep in mind that although the contributed Drupal modules are free of cost, they are
not feather light. Unused modules can unnecessarily consume a lot of space and resources
which can make your Drupal website heavier and slower. So, before you download a module,
analyze if you really need it or if any other core module can perform the same functionality for
your website. Also, feel free to abandon those hardly used and inactive modules to make your
website feel healthier and light.
It is very important to choose modules that are actively maintained, updated and published by
the developers. Because, if you run into any security vulnerabilities or any other issues, you
know that you can get a quicker response from the developer/contributor. Also, you will be
sure that an updated new version is on its way soon.
Using popular modules means you can trust the module to do the job and be secure with less
issues. In the module’s project page on drupal.org, you can see the number of downloads and
how many websites are currently using that particular module.
On the right side of the Drupal module’s project page, you can find the number of issues and
bugs (open and closed) associated with that module. On clicking on the link, you will be able to
see a detailed report of all bugs and issues. Looking at this can give you an idea if the module
can really help you with your functionality or not. The ‘Last Updated’ date can also give you an
idea of how active the developers are in solving issues and how responsive they are.
A Drupal website’s functionalities are extended and boosted with the help of the various
modules that are contributed by Drupal’s open-source community. Making the right choice of
Drupal 8 modules that needs to do exactly what you want is a daunting task. We are a Drupal
Development Company with a highly experienced team of Acquia certified developers who can
help you make the right choice and enhance the capabilities of your Drupal website. Contact
us for more information.
Leave us a CommentShefali ShettyApr 05, 2017 Recent Posts Image Top Drupal 8 Modules You Absolutely Need For your next Drupal project Image Why is Drupal CMS the top choice for Government Websites in 2020? Image An Introduction to PHP Standard Recommendation (PSR) Want to extract the maximum out of Drupal? Contact us TALK TO US Featured Success Stories
Know more about our technology driven approach to recreate the content management workflow for 7.ailink
Find out how we transformed the digital image of world’s largest healthcare provider, an attribute that defined their global presence in the medical world.link
Develop an internal portal aimed at encouraging sellers at Flipkart to obtain latest insights with respect to a particular domain.link