May 232014
 

so, i recently ran into this frustratingly undescriptive error:

[root@puppet environments]# puppet agent -t
Info: Retrieving plugin
Info: Loading facts in /etc/puppetlabs/puppet/modules/concat/lib/facter/concat_basedir.rb
Info: Loading facts in /opt/puppet/share/puppet/modules/stdlib/lib/facter/facter_dot_d.rb
Info: Loading facts in /opt/puppet/share/puppet/modules/stdlib/lib/facter/puppet_vardir.rb
Info: Loading facts in /opt/puppet/share/puppet/modules/stdlib/lib/facter/pe_version.rb
Info: Loading facts in /opt/puppet/share/puppet/modules/stdlib/lib/facter/root_home.rb
Info: Loading facts in /opt/puppet/share/puppet/modules/concat/lib/facter/concat_basedir.rb
Info: Loading facts in /opt/puppet/share/puppet/modules/pe_common/lib/facter/windows.rb
Info: Loading facts in /opt/puppet/share/puppet/modules/postgresql/lib/facter/postgres_default_version.rb
Info: Loading facts in /opt/puppet/share/puppet/modules/auth_conf/lib/facter/custom_auth_conf.rb
Info: Loading facts in /opt/puppet/share/puppet/modules/firewall/lib/facter/iptables_persistent_version.rb
Info: Loading facts in /opt/puppet/share/puppet/modules/firewall/lib/facter/iptables_version.rb
Info: Loading facts in /opt/puppet/share/puppet/modules/firewall/lib/facter/ip6tables_version.rb
Info: Loading facts in /opt/puppet/share/puppet/modules/pe_puppetdb/lib/facter/puppetdb_server_status.rb
Info: Loading facts in /var/opt/lib/pe-puppet/lib/facter/iptables_persistent_version.rb
Info: Loading facts in /var/opt/lib/pe-puppet/lib/facter/facter_dot_d.rb
Info: Loading facts in /var/opt/lib/pe-puppet/lib/facter/custom_auth_conf.rb
Info: Loading facts in /var/opt/lib/pe-puppet/lib/facter/iptables_version.rb
Info: Loading facts in /var/opt/lib/pe-puppet/lib/facter/puppetdb_server_status.rb
Info: Loading facts in /var/opt/lib/pe-puppet/lib/facter/concat_basedir.rb
Info: Loading facts in /var/opt/lib/pe-puppet/lib/facter/puppet_vardir.rb
Info: Loading facts in /var/opt/lib/pe-puppet/lib/facter/windows.rb
Info: Loading facts in /var/opt/lib/pe-puppet/lib/facter/pe_version.rb
Info: Loading facts in /var/opt/lib/pe-puppet/lib/facter/root_home.rb
Info: Loading facts in /var/opt/lib/pe-puppet/lib/facter/ip6tables_version.rb
Info: Loading facts in /var/opt/lib/pe-puppet/lib/facter/postgres_default_version.rb
Error: Could not retrieve catalog from remote server: Error 400 on SERVER: (<unknown>): mapping values are not allowed in this context at line 5 column 9
Warning: Not using cache on failed catalog
Error: Could not retrieve catalog; skipping run

after some head-scratching and finger-pointing, we narrowed it down to faulty yaml files in /var/lib/hiera. we had just recently set up hiera, and the developer working on populating it needed a quick lesson in yaml syntax.

hopefully this helps someone else demystify this cryptic error.

Nov 122013
 

something i feel could greatly help someone else in our situation this morning: a potential fix for mysql errno 1236.

131112  9:26:02 [Note] Slave I/O thread: connected to master 'repl@x.x.x.x:3306',  replication started in log 'je.000041' at position 90217258
131112  9:26:02 [ERROR] Error reading packet from server: Could not find first log file name in binary log index file ( server_errno=1236)
131112  9:26:02 [ERROR] Got fatal error 1236: 'Could not find first log file name in binary log index file' from master when reading data from binary log
131112  9:26:02 [Note] Slave I/O thread exiting, read up to log 'je.000041', position 90217258

after struggling for close to 30 minutes to find a fix – purging binary logs, checking index files, using alternate binlogs – we had yet to come up with a fix. what could be going wrong?

it turned out to be really simple, and really dumb. we had typo’d the master_log_file attribute of our change master – instead of je.000041, it should have read je1.000041!

changing the binlog name in the change master statement fixed everything. what a pain..