Feed aggregator

groups.drupal.org frontpage posts: Let's fix critical Drupal 8 issues together!

Drupal News - October 20, 2014 - 7:57am

Every Friday at noon Pacific (3pm New York, 9pm Berlin, 6am Saturday in Sydney) I will be in #drupal-contribute helping people fix critical issues. I will prepare 2-3 issues with up to date, actionable issue summaries, familiarize myself with the problems and the suggested solution in the issue so that I can answer questions.

If you're someone who has already worked some in the Drupal.org issue queue (so are familiar with patches, coding standards, etc.), even if your experience is not in the core queue, please come by! It's helpful if you know something of Drupal 8 as well, but not necessary.

If you're new to contributing to Drupal in general, you can go to https://www.drupal.org/core-mentoring for a session or two to learn the skills you need to fix critical issues. If you're new to Drupal 8, https://api.drupal.org/api/drupal/8 is a great starting point.

Hope to see you there!

Phase2: Simplify Your Logstash Configuration

Drupal News - October 20, 2014 - 6:16am

As I mentioned in my recent post, I got a chance to upgrade the drupal.org ELK stack last week. In doing so, I got to take a look at a Logstash configuration that I created over a year ago, and in the course of doing so, clean up some less-than-optimal configurations based on a year worth of experience and simplify the configuration file a great deal.

The Drupal.org Logging Setup

Drupal.org is served by a large (and growing) number of servers. They all ship their logs to a central logging server for archival, and around a month’s worth are kept in the ELK stack for analysis.

Logs for Varnish, Apache, and syslog are forwarded to a centralized log server for analysis by Logstash. Drupal messages are output to syslog using Drupal core’s syslog module so that logging does not add writes to Drupal.org’s busy database servers. (@TODO: Check if these paths can be published.) Apache logs end up in/var/log/apache_logs/$MACHINE/$VHOST/transfer/$DATE.log, Varnish logs end up in/var/log/varnish_logs/$MACHINE/varnishncsa-$DATE.log and syslog logs end up in /var/log/HOSTS/$MACHINE/$DATE.log. All types of logs get gzipped 1 day after they are closed to save disk space.

Pulling Contextual Smarts From Logs

The Varnish and Apache logs do not contain any content in the logfiles to identify which machine they are from, but the file input sets a path field that can be matched with grok to pull out the machine name from the path and put it into the logsource field, which Grok’s SYSLOGLINE pattern will set when analyzing syslog logs.

Filtering on the logsource field can be quite helpful in the Kibana web UI if a single machine is suspected of behaving weirdly.

Using Grok Overwrite

Consider this snippet from the original version of the Varnish configuration. As I mentioned in my presentation, Varnish logs are nice in that they inclue the HTTP Host header so that you can see exactly which hostname or IP was requested. This makes sense for a daemon like Varnish which does not necessarily have a native concept of virtual hosts (vhosts,) whereas nginx and Apache default to logging by vhost.

Each Logstash configuration snippet shown below assumes that Apache and Varnish logs have already been processed using theCOMBINEDAPACHELOG grok pattern, like so.

filter { if [type] == "varnish" or [type] == "apache" { grok { match => [ "message", "%{COMBINEDAPACHELOG}" ] } } }

The following snippet was used to normalize Varnish’s request headers to not include https?:// and the Host header so that therequest field in Apache and Varnish logs will be exactly the same and any filtering of web logs can be performed with the vhost andlogsource fields.

filter { if [type] == "varnish" { grok { # Overwrite host for Varnish messages so that it's not always "loghost". match => [ "path", "/var/log/varnish_logs/%{HOST:logsource}" ] } # Grab the vhost and a "request" that matches Apache from the "request" variable for now. mutate { add_field => [ "full_request", "%{request}" ] } mutate { remove_field => "request" } grok { match => [ "full_request", "https?://%{IPORHOST:vhost}%{GREEDYDATA:request}" ] } mutate { remove_field => "full_request" } } }

As written, this snippet copies the request field into a new field called full_request and then unsets the original request field and then uses a grok filter to parse both the vhost and request fields out of that synthesized full_request field. Finally, it deletesfull_request.

The original approach works, but it takes a number of step and mutations to work. The grok filter has a parameter calledoverwrite that allows this configuration stanza to be considerably simlified. The overwrite paramter accepts an array of values thatgrok should overwrite if it finds matches. By using overwrite, I was able to remove all of the mutate filters from my configuration, and the enture thing now looks like the following.

filter { if [type] == "varnish" { grok { # Overwrite host for Varnish messages so that it's not always "loghost". # Grab the vhost and a "request" that matches Apache from the "request" variable for now. match => { "path" => "/var/log/varnish_logs/%{HOST:logsource}" "request" => "https?://%{IPORHOST:vhost}%{GREEDYDATA:request}" } overwrite => [ "request" ] } } }

Much simpler, isn’t it? 2 grok filters and 3 mutate filters have been combined into a single grok filter with two matching patterns and a single field that it can overwrite. Also note that this version of the configuration passes a hash into the grok filter. Every example I’ve seen just passes an array to grok, but the documentation for the grok filter states that it takes a hash, and this works fine.

Ensuring Field Types

Recent versions of Kibana have also gotten the useful ability to do statistics calculations on the current working dataset. So for example, you can have Kibana display the mean number of bytes sent or the standard deviation of backend response times (if you are capturing them – see my DrupalCon Amsterdam slides for more information on how to do this and how to normalize it between Apache, nginx, and Varnish.) Then, if you filter down to all requests for a single vhost or a set of paths, the statistics will update.

Kibana will only show this option for numerical fields, however, and by default any data that has been parsed with a grok filter will be a string. Converting string fields to other types is a much better use of the mutate filter. Here is an example of converting the bytes and the response code to integers using a mutate filer.

@TODO: Test that hash syntax works here!

filter { if [type] == "varnish" or [type] == "apache" { mutate { convert => { [ "bytes", "response" ] => "integer", } } } }

Lessons Learned

Logstash is a very powerful tool, and small things like the grokoverwrite parameter and the mutate convert parameter can help make your log processing configuration simpler and result in more usefulness out of your ELK cluster. Check out Chris Johnson’s post about adding MySQL Slow Query Logs to Logstash!

If you have any other useful Logstash tips and tricks, leave them in the comments!

 

4Sitestudios.com Drupal Blog: Major Drupal 7 Security Vulnerability - Update Now!

Drupal News - October 20, 2014 - 6:13am

Last week the Drupal security team announced the existence of a major security vulnerability in all versions of Drupal 7. This vulnerability is rated as “highly critical” because it allows an attacker to take full control of your site remotely, without needing to log in as a privileged user. Attacks using this vulnerability are already being reported.

If 4Site built your Drupal 7 site, if we handle your site maintenance, or if you're just looking for someone to help you apply the update on your site and keep your site secure, please contact us!

Bluespark Labs: Uninstalling and purging field modules all at once

Drupal News - October 20, 2014 - 3:14am

Sometimes we want to uninstall a module from our Drupal site but we can't do it because we get this dependency: "Required by: Drupal (Field type(s) in use - see Field list)". Even if you delete the fields provided by the module via the UI or programmatically by executing field_delete_field() function you will get a new dependency "Required by: Drupal (Fields pending deletion)".

These dependencies are created by Drupal core to avoid that a module is uninstalled until all the data related to its fields is removed from the database, in order to maintain consistency.

This has several drawbacks, the first one being that you can't uninstall your module when you want, and you have to wait until all the field data values are removed from the database (The rather strangely named field_deleted_data_XX and field_deleted_revision_XX tables) and the meta-information stored in field_config and field_config_instance tables is removed. And most importantly, nobody actually knows when this is going to happen! These database rows are removed in batches on each cron task execution. So depending on our cron regularity and the amount of data stored in our field tables, this tasks can last for minutes to weeks.

This is a problem because, naturally, we want to uninstall our module now and not be forced to check periodically our production database to see if we are allowed to uninstall the module once all that information has been removed from the database.

To avoid such situations and regain control, you can perform all these tasks in a hook_update_N() function, forcing the deletion of all the information and finally uninstalling the module. You can check the code in the gist below:

The job is divided in three parts: The data definition, field data purge and module list clean.

In the data definition task we provide all the required data we need to perform the task, the name of the field to delete, and given that information, we get the field_info array and the name of the module to be uninstalled. Finally, field_delete_field() is executed.

After that the field data is purged in the batch body, and since we don't know how much data we will have to purge, we remove just 100 database rows per batch execution. After each purge we check if all the data has been removed to decide if we have to remove more data from the database or continue to the final part.

Once all the data and metadata related to the module is removed from the database, the Drupal field types dependency is gone and we are granted the ability to disable and uninstall our module cleanly. Finally, we can drop the empty field_deleted_data_XX and field_deleted_revision_XX tables to keep clean our database.

Using this approach, we have two key benefits: a. we are sure that the module is disabled and our database is clean, and b. we are confident that we can remove the module from our repository, given that in the next deploy we won't get any dependency conflict with that module.

Tags: Drupal Planet

Visitors Voice: That is why we sponsor the Search API Solr module

Drupal News - October 20, 2014 - 2:03am
Since june 2014 we sponsor the Search API Solr module. There are no strings attached, and we sponsor the maintainer Thomas Seidl a.k.a Drunken Monkey with a couple of hours every month that he can spend as he likes. It could be bug fixing, features asked for or working on the Drupal 8 version. We […]

Gizra.com: Gizra - We've Got Your Headless Covered

Drupal News - October 18, 2014 - 1:00pm
What's the name of the Angular component for login?

The difficulties in creating a semi or fully decoupled site isn't in the RESTful part. Spitting out JSON is now covered by several modules, including RESTful which aims for a "best practices" solution.

One of the real problems, though, is how to prevent us, the community, from re-inventing the wheel over and over again. Basically, how do we package our frontend code similarly to how we package our generic backend code - AKA "modules". I discussed these problems, and offered some solutions in my "BoF" persentation:

Continue reading…

Doug Vann: Drupal Training at Drupal Camps And Why We Need More Of It

Drupal News - October 17, 2014 - 5:41pm

Drupal Camp Road Warrior
By the end of 2014, I will have hit 50 Drupal Camps! It took 72 months to hit 22 cities, in 16 states! In that time, I've seen Drupal Camps run in almost every conceivable way possible. From Madison WI to Orlando FL, from NewYork NY to San Diego CA, I've seen thousands of attendees flocking to these events, all with the hopes of growing in their knowledge and understanding of Drupal. In my experience, the system works -- mostly.
But, we can do better.

We all know the drill
You assemble a bunch of speakers. They will deliver a bunch of sessions. You try to group these sessions into tracks, if you can. You wrestle with how to add a few sessions about the Drupal Community or maybe about Business or a few odd sessions that don't fit into your tracks. Oh yah... You almost forgot about the beginners, so you have a session or two that demystifies one topic or another.

The N00B experience
You would be surprised at how many people show up to a Drupal Camp who don't know what a node is. Or if they do know what a node is, they don't know how to create their own content types. Or if they do know how to create content types, they don't know how to create Views. These people show up and attend sessions that they have little chance of comprehending. They sit down for up to an hour per session listening to senior developers from major Drupal shops talk about nodes and fields and blocks and views-displays and modules. The whole time they may be thinking, "Dang! I thought by showing up for a day or two I would start picking this stuff up!?" But they're not.

Meet the N00Bs
Who are these people who are "New To Drupal?" Well, for starters, they're probably not really that new to Drupal! Based on my experiences, here is an incomplete list of ppl who regularly attend my classes.

  1. Certainly anyone who just discovered Drupal very recently and has come to the camp to gain a better understanding of Drupal. [This is not always the biggest portion]
  2. Individuals who have been to a couple camps and have tried to read the books or watch the videos but still haven't had the needed "AHA!" moments to grasp it all.
  3. Individuals who work for a University or Government or Company who uses, or is considering, Drupal. [This is a BIG ONE]
    • People, often with other web skills [sys admins, java, asp, php, etc] who are sent by their employers to scope out Drupal and/or to learn how to use it.
    • People coming to gain skills in an effort to alleviate their, or their employer's, dependency on vendors. [This happens a lot!]
    • New hires to Drupal shops or Design shops or shops offering web related services who are looking to better provide Drupal related services. 
    • People who know plenty, but want to make sure they are properly grounded.
    • People who come in the hopes of asking lots of questions!

I've seen all that and more. Multitudes of people are coming to camps in hopes of really wrapping their minds around how Drupal solves the modern problem of publishing dynamic content on the web. Too often, without a day of training they leave the camp with the same [and more] questions than they arrived with.

What they really want/need
After attending camp after camp, it's a proven fact. People are coming to learn what Drupal is and how to use it.  If the camp has no full day training opportunity then many are going to drown in the other sessions and simply not get what they really need.
I'll just be frank at this point. I believe that every camp needs to have a full day of beginner training. I believe that this training should be delivered not across differing tracks with differing speakers, but by the same individual, or group of individuals, working together to provide the full day of training. I have done this time and time again and I see the relief on people's faces as they gain a practical understanding of the power and flexibility of Drupal and how they can leverage it. This day of training starts them down the road of really learning Drupal. If there's a 2nd day of camp, I can PROMISE you that they will get far more out of it after a day of training.

How to provide a day of training at a Drupal Camp
There are many ways! Here's a list that is, by no means, exhaustive.

  1. Some camps have a dedicated day just for trainings on the day before the regular camp.
    • This is effecive not only for beginner classes but for classes on SEO, Drupal 8, Module Development, etc.
    • Most often training takes place in the same location as the camp, but occasionally it is not.
  2. Some camps simply reserve one track and dedicate it to a full day of training.
    • I've done this quite a few times where I have a room all day while others hop from session to session.
    • This is easier if you can't dedicate a whole day to training.
  3. The content in the full day Drupal beginner's training.
    • In some camps someone leads the class through the Acquia curriculum of Drupal In A Day
    • Some camps have a vendor come in and do the training
      • Doug Vann! If you want me to join your camp and present a day of training call me at 765-5-DRUPAL or CONTACT ME
      • I've seen posts from BLINK REACTION & OSTRAINING about their various full day offerings at Drupal Camps as well.
      • If I missed anyone who has travelled to multiple camps and provided full day trainings in the past and would do so again, leave a comment and I'll add you here. :-)
    • Some camps have used the BuildAModule.com Mentored Training method.
  4. The finances of a full day of training. Here's how I've experienced this as a trainer.
    • Some camps offer it for free or as part of the Camp fee that attendees have already paid.
    • Some camps charge attendees enough to cover the cost of catering.
    • Some camps charge a flat fee per attendee and share a percentage with the trainer.
    • Some camps procure a "training sponsor" and hand that sum off to the trainer.

Conclusion
Every Drupal Camp can do this! I've been invited to one-day camps and they give me one of their rooms for the whole day. I show up and deliver the full day of Drupal Beginner Training. Sadly, I never get to see any of the other sessions. Oh well... After 50 Drupal Camps, I've seen plenty of Drupal Sessions! :-)
Providing a full day of training will definitely raise your attendance. Universities, Governments, and Companies will send people. People will ask their employers to send them. Sponsors will really appreciate the fact that you're providing extra value to a broader audience.
Seriously folks... What more can I say? 

Full Day Trainings at Drupal Camps is a Big Win for everyone involved!

 

 

Drupal Planet

View the discussion thread.

Syndicate content