Camelia, the Perl 6 bug

IRC log for #gluster, 2012-12-04

| Channels | #gluster index | Today | | Search | Google Search | Plain-Text | summary

All times shown according to UTC.

Time Nick Message
00:12 nightwalk joined #gluster
00:15 hattenator joined #gluster
00:27 dalekurt_ joined #gluster
00:31 nightwalk joined #gluster
00:36 Eco1 joined #gluster
00:44 nightwalk joined #gluster
00:44 diegol joined #gluster
00:50 mooperd_ joined #gluster
01:01 nightwalk joined #gluster
01:16 genewitch what happens if gluster has one volume running replica and i need to restart it to create another volume, does that break the original volume?
01:16 genewitch or should i stop the existing volume before i restart glusterd
01:16 Eco1 genewitch: why do you need to stop it?
01:16 genewitch because it says the prefix was already part of a volume
01:17 genewitch :-)
01:17 JoeJulian Paste the actual error message
01:17 JoeJulian verbatim
01:17 genewitch /gluster/ebs or a prefix of it is already part of a volume
01:17 glusterbot genewitch: To clear that error, follow the instructions at http://goo.gl/YUzrh or see this bug http://goo.gl/YZi8Y
01:17 genewitch yeah i am trying to, bot.
01:17 JoeJulian hehe
01:17 genewitch :-D
01:17 Eco1 genewitch: just do it!  ;)
01:18 genewitch without stopping the existing volume?
01:18 JoeJulian Are you trying to put the same brick in two different volumes?
01:19 genewitch no, i've got two seperate bricks on each server, i want two seperate volumes. one's allegedly faster, i want to see by how much
01:19 Eco1 genewitch: JoeJulian there is a script for this now i think
01:19 Eco1 glusterfs/extras/backend-xattr-sanitize.sh
01:19 Eco1 haven't tested it but the name looks promising
01:20 JoeJulian genewitch: Are you reusing a brick that you've used previously?
01:20 genewitch JoeJulian: yes
01:20 JoeJulian And that brick is no longer part of a volume?
01:21 genewitch correct
01:21 JoeJulian Then you don't need to worry about glusterd.
01:21 genewitch so i've sanitized the brick paths on both servers
01:22 JoeJulian I love using unicorn for running ruby stuff. It's fun to type, "killall unicorn"
01:22 genewitch wooooot
01:42 nightwalk joined #gluster
01:42 kevein joined #gluster
01:49 lng joined #gluster
01:50 lng JoeJulian: Hi! Short question about split brain
01:50 lng JoeJulian: can I also delete those entries like <gfid:b8fc528a-6f45-48a3-83da-4f35f3b63662>?
01:55 nightwalk joined #gluster
01:58 mooperd joined #gluster
02:32 sunus joined #gluster
02:33 nightwalk joined #gluster
02:44 sunus hi, what is vdsm-glusterfs doing?
02:45 Technicool managing gluster volumes via ovirt
02:45 sunus can i have separate vdsm and glusterfs installation?
02:46 Technicool probably, but not likely if you want to manage gluster via ovirt
02:46 sunus Technicool: i am not sure is vdsm-glusterfs is a TOOL to put two together or a package containing them?
02:46 Technicool if you wanted to have them separate for some reason, it would guess it would work
02:47 sunus Technicool: so, it's a pre-merged package containing ovirt and glusterfs?
02:47 Technicool vdsm is the management portion of ovirt i think
02:47 Technicool via the ovirt web ui, afaik
02:47 Technicool doesn't contain gluster but lists it as a dep
02:49 Technicool there was also a bug previously with ovirt using Gluster NFS. so if you plan on using Gluster to store virtual disk images you at least need to make sure you are connecting via fuse client
02:50 sunus Technicool: Thank you!! i didn't know that previously. and i was wondering what is that bug you mention?
02:51 Technicool sunus, when using the Gluster NFS within oVirt, you commonly get lock errors
02:51 Technicool not sure if that has been addressed yet or not
02:56 sunus Technicool: i am notice that, in fact, my company is about to using it
02:57 Technicool sunus, via ovirt or Gluster by itself?
02:57 sunus sunus: gluster via ovirt
02:58 sunus Technicool: so, i am do researchs in vdsm-gluster
02:58 Technicool in that case, is there a reason not to let it be managed from ovirt?
02:58 sunus Technicool: not sure yet
02:58 sunus Technicool: :(
02:59 Technicool sunus, if there isn't a compelling reason you could just make things easy on yourself  ;)
02:59 sunus Technicool: okay, thank you! our plan is to make the very basic function work this week, so, i will let you know:)
03:12 nightwalk joined #gluster
03:26 nightwalk joined #gluster
03:51 sgowda joined #gluster
03:55 lh joined #gluster
03:55 lh joined #gluster
03:56 hagarth joined #gluster
03:57 bharata joined #gluster
04:10 nightwalk joined #gluster
04:22 sripathi joined #gluster
04:28 raghu joined #gluster
04:29 sgowda joined #gluster
04:31 hagarth joined #gluster
04:38 nightwalk joined #gluster
04:40 sunus Technicool: i met the locks...
05:06 nightwalk joined #gluster
05:10 vpshastry joined #gluster
05:19 nightwalk joined #gluster
05:22 bala joined #gluster
05:26 chacken joined #gluster
05:30 mohankumar joined #gluster
05:30 sgowda joined #gluster
05:43 manik joined #gluster
05:45 vijaykumar joined #gluster
05:49 ankit9 joined #gluster
05:55 net_list_5 joined #gluster
05:56 vimal joined #gluster
05:58 nightwalk joined #gluster
06:08 ramkrsna joined #gluster
06:08 ramkrsna joined #gluster
06:10 ffog joined #gluster
06:21 deepakcs joined #gluster
06:22 ankit9 joined #gluster
06:32 harshpb joined #gluster
06:36 vijaykumar joined #gluster
06:37 shireesh joined #gluster
06:44 nightwalk joined #gluster
06:46 sunus Technicool: hi, are you online?
06:47 sunus why i can create volume via ovirt, but i can not start the volume?
06:57 puebele1 joined #gluster
07:00 overclk joined #gluster
07:05 nightwalk joined #gluster
07:10 inodb_ joined #gluster
07:16 stre10k joined #gluster
07:18 ngoswami joined #gluster
07:22 sunus how can i configure ovirt with gluster ? any doc?
07:22 Technicool sunus, most likely you will get answers in the ovirt channel on OFTC
07:23 Technicool irc.oftc.net
07:24 Technicool there are a few docs written by jbrooks as well
07:25 Technicool sunus - http://www.gluster.org/category/ovirt/
07:25 glusterbot Title: oVirt 3.1, Glusterized | Gluster Community Website (at www.gluster.org)
07:28 sunus Technicool: i am reading that
07:28 sunus Technicool: kinda confusing, i gotta say:(
07:29 Technicool sunus, yeah, the best doc i have seen for it is the RHEV beta doc
07:29 Technicool for me, i had issues doing the single node install, things got much easier when i used one node for management and others to house ovirt nodes
07:34 guigui1 joined #gluster
07:34 sunus Technicool: do i need to create a HOST in ovirt engine first ? i failed hours ago, then i deleted all the hosts..
07:34 inodb joined #gluster
07:34 Technicool once the management engine is installed, create a DC, then cluster, then host, but the host should be on another node than the management one
07:39 ankit9 joined #gluster
07:41 sunus Technicool: i now add a host to be approved, after approve, i add a dc, right?
07:42 Technicool sunus, thought it was backwards from that, but if it works, then, yes ;)
07:42 sunus Technicool: approve operation needs a data center and host cluster, which i didn't create yet.
07:42 inodb_ joined #gluster
07:42 sunus Technicool: so now, i have to create a DC and host cluster. right?
07:43 Technicool sunus, thats how i recall doing it, yes
07:43 sunus Technicool: haha, thank you! let me give it a go
07:43 sunus Technicool: let you know if any bad happens:(
07:44 Technicool sunus, you should hop over to the #ovirt room on oftc, i am about to drop off, early meetings tomorrow
07:44 sunus Technicool: New data center with type NFS with ver 3.1
07:44 sunus Technicool: ok:( you are more than helpful:)
07:44 ctria joined #gluster
07:46 hagarth joined #gluster
07:48 dobber joined #gluster
07:48 nightwalk joined #gluster
07:50 dobber_ joined #gluster
07:55 puebele joined #gluster
07:55 puebele left #gluster
07:59 lkoranda joined #gluster
07:59 andreask joined #gluster
08:00 Azrael808 joined #gluster
08:06 gbrand_ joined #gluster
08:20 tjikkun_work joined #gluster
08:23 inodb joined #gluster
08:27 inodb_ joined #gluster
08:31 nightwalk joined #gluster
08:35 yinyin joined #gluster
08:42 tryggvil_ joined #gluster
08:42 tryggvil joined #gluster
08:47 gbrand_ joined #gluster
08:50 jmara joined #gluster
08:57 toruonu joined #gluster
08:59 nightwalk joined #gluster
09:00 stre10k What is more suitable for HA storage of 3+TB pictures of different sizes to distribute their by web server. Gluster mirror or DRBD?
09:01 sunus i add glusterfs volume into ovirt, got error:Error: Cannot add Storage. Internal error, Storage Connection doesn't exist.
09:04 hackez joined #gluster
09:06 Staples84 joined #gluster
09:20 yinyin joined #gluster
09:21 TheHaven joined #gluster
09:25 nightwalk joined #gluster
09:32 gbrand_ joined #gluster
09:32 H__ Any ideas why syncdaemon/libcxattr.py : getattr(cls.libc, syscall)(*((path,) + a + (buf, siz)))  returns -1 ->  http://fpaste.org/iFaj/ ?
09:32 glusterbot Title: Viewing gsyncd crash by Hans (at fpaste.org)
09:33 ndevos "Cannot allocate memory" might indicate that the gsyncd process does not have enough memory?
09:37 H__ i highly doubt that, the process never was big in RAM, and the box has enough spare ram. It's only function is gluster, it has 16 GiB ram.
09:48 sripathi joined #gluster
09:49 mooperd joined #gluster
09:56 ramkrsna joined #gluster
09:56 mgebbe_ joined #gluster
10:06 nightwalk joined #gluster
10:09 olisch joined #gluster
10:09 sripathi1 joined #gluster
10:10 Alpinist joined #gluster
10:11 yinyin joined #gluster
10:13 ndevos H__: and the user running gsyncd does not have any limits on memory? like /etc/security/limits.conf
10:15 H__ ndevos: right, it's running as root even.
10:18 jmara joined #gluster
10:20 Alpinist Good morning
10:21 Alpinist Anyone having an idea why mount other filesystems while rebooting hangs, after doing a yum update
10:22 Alpinist Sending an ctrl+alt_del, rebooting for the seconds time solves it
10:25 layer3switch joined #gluster
10:25 glusterbot New news from resolvedglusterbugs: [Bug 844324] core: possible memory leak <http://goo.gl/XqVTp> || [Bug 860114] glusterfsd crash <http://goo.gl/MLnjd> || [Bug 768330] Memory leakage in brick process.[Release-3.3.qa15 <http://goo.gl/OVM46>
10:31 pkoro joined #gluster
10:32 nightwalk joined #gluster
10:40 olisch alpinist:i have the same problem on centos nodes
10:40 olisch did you use _netdev as mount option?
10:40 olisch this might/should help, but not in my case
10:40 Alpinist yes, it only occurs after an update
10:40 olisch still wondering why
10:41 Alpinist i'm try to find it out, looks like something after a kernel update
10:41 olisch ooh, might be that we have done an update before rebooting, too
10:42 Alpinist we are running version 3.2
10:42 olisch we, too
10:43 puebele1 joined #gluster
10:44 olisch we couldnt reproduce the problem, it only happened after general maintenance and then on several hundred nodes, because we have always done a yum update during maintenance and also installed kernel updates, this might really be an issue
10:44 Alpinist i'm now trying to first umount, stop and than doing an upgrade
10:44 hagarth1 joined #gluster
10:45 olisch i dindt thought about the update aspect yet
10:45 Alpinist :) we had the same... on a nice night, doing some upgrades ....
10:47 nightwalk joined #gluster
10:47 Alpinist unmounting and stopping glusterfs before updating didn't work
10:48 Alpinist after day's i now know how to trigger the hanging
10:48 joeto joined #gluster
10:49 mooperd joined #gluster
10:51 Alpinist did you also saw "No kdump initial ramdisk found" and than rebuilding /boot/ini.....
10:52 olisch i dont know, my colleagues did the maintenance and just reported the problem and the fun of rebooting all nodes via ipmi for a second time ;)
10:53 olisch but with a new kernel and kdump installed, i think this should be normal behaviour
10:54 olisch just looked on our nodes, no kdump installed
10:54 olisch so we wont have had that message
10:55 Alpinist one test less to do :) thanks
10:55 olisch how can you trigger the hanging? just by "yum update", also when there are no updates to be installed?
10:55 olisch or is a kernel change needed?
10:56 Alpinist a yum update without installing anything and than rebooting didn't trigger the hanging
10:56 Alpinist i'll try by just installing something, but not the kernel
11:00 Alpinist oooh this is strang: i did yum install telnet, reboot and triggered the hanging
11:01 nightwalk joined #gluster
11:02 Alpinist can it be that's it nog gluster but all networkmounts?
11:03 duerF joined #gluster
11:04 duerF joined #gluster
11:07 puebele joined #gluster
11:09 diegol joined #gluster
11:09 olisch yes, might be all networkmounts, but we are using gluster only
11:11 yinyin joined #gluster
11:12 diegol hey anyone with a distributed volume and a very high ram usage on just one of them?
11:12 olisch not on just one of them
11:12 olisch but when you enable nfs, which is enabled by default
11:12 olisch then you have high ram usage
11:13 olisch high ram usage on client or server side?
11:14 raghu joined #gluster
11:16 sunus http://www.soimort.org//posts/125/index.html
11:16 glusterbot Title: Mort | 依赖性地狱和Arch的哲♂学 (at www.soimort.org)
11:16 diegol server side olisch
11:16 sunus sorry
11:16 diegol and yeah my clients use nfs
11:16 olisch well, then you can't disable nfs
11:17 olisch but that was something that i recognized, enabling nfs has eaten about 4gig of ram
11:17 diegol this is eating about 14.3gig
11:17 diegol just on one of the two nodes
11:18 diegol is this normal? that all the work is being carried but just one of the nodes?
11:18 diegol *by
11:18 vijaykumar joined #gluster
11:19 olisch just using the native gluster clients, but as i remember, when using nfs the gluster volume is mounted on a node and then reexported via nfs, so this node has to do more
11:20 olisch all traffic has to pass this node
11:20 diegol the thing is I need to use nfs due to some of my users work with small files
11:21 diegol and the native glusterfs client gives me poor performance
11:21 diegol some processes enter D state and it was a pain
11:21 olisch then the ram usage might be related to some nfs caching? i'm not experienced with nfs, just heard about it
11:22 diegol don't know yet
11:22 diegol I noticed this yesterday
11:22 diegol I will perform some tests when user's experiments end
11:27 sgowda joined #gluster
11:29 overclk joined #gluster
11:37 diegol seems like you are right olisch :-)
11:39 ndevos olisch, AlpinistLunch: maybe yout netfs service is started before glusterd? if so, the volumes would not be available for mounting yet
11:39 dobber joined #gluster
11:40 nightwalk joined #gluster
11:44 puebele joined #gluster
11:50 mooperd joined #gluster
11:52 mooperd joined #gluster
11:55 harshpb joined #gluster
11:56 nightwalk joined #gluster
12:06 vijaykumar left #gluster
12:13 sripathi joined #gluster
12:18 kkeithley1 joined #gluster
12:21 overclk joined #gluster
12:27 nightwalk joined #gluster
12:28 Alpinist ndevos in that case it would be hanging every time i think
12:36 Alpinist joined #gluster
12:36 clag_ joined #gluster
12:37 Alpinist did i mis something?
12:43 lh joined #gluster
12:43 lh joined #gluster
12:46 olisch no
12:51 inodb joined #gluster
13:01 nightwalk joined #gluster
13:02 guigui4 joined #gluster
13:02 aliguori joined #gluster
13:05 berend` joined #gluster
13:06 lkoranda joined #gluster
13:07 gbrand_ joined #gluster
13:07 Alpinist so the big question is: what does yum change when doing an upgrade or install, that gluster or netmount uses
13:08 olisch i am just trying to reproduce, but don't get it hanging
13:09 Alpinist what os are you using?
13:09 Alpinist i'm testing on a SL 6.2
13:10 stre10k left #gluster
13:10 olisch it is an installation of parallels bare metal which uses cloud os which is mainly a normal centos 6
13:12 Alpinist 6.3 or lower?
13:12 olisch 6.1
13:17 Alpinist A reinstall of a package also triggers it
13:18 gbrand__ joined #gluster
13:28 lkoranda joined #gluster
13:30 the-me joined #gluster
13:31 shireesh joined #gluster
13:32 nightwalk joined #gluster
13:37 shireesh joined #gluster
13:43 ekuric joined #gluster
13:44 ekuric joined #gluster
13:47 gbr joined #gluster
13:52 gbr joined #gluster
13:52 bala joined #gluster
13:55 shireesh joined #gluster
14:01 inodb_ joined #gluster
14:06 nightwalk joined #gluster
14:19 nightwalk joined #gluster
14:24 guigui1 joined #gluster
14:27 bennyturns joined #gluster
14:36 nightwalk joined #gluster
14:41 yinyin joined #gluster
14:45 gbr joined #gluster
14:45 gbr My gluster servers are going insane.  My VM's are toast every morning.  These things have been running great since June, and now they are flakey as heck.
14:45 gbr [2012-12-04 08:44:41.710387] E [nfs3.c:2163:nfs3_write_resume] 0-nfs-nfsv3: Unable to resolve FH: (192.168.20.12:769) NFS_RAID6_FO : 7236b1f6-380d-419c-9bbc-fee9f6fc63e2
14:45 gbr [2012-12-04 08:44:41.710421] W [nfs3-helpers.c:3389:nfs3_log_common_res] 0-nfs-nfsv3: XID: ccdeef32, WRITE: NFS: 2(No such file or directory), POSIX: 14(Bad address)
14:47 UnixDev joined #gluster
14:52 stopbit joined #gluster
14:56 toruonu joined #gluster
15:01 __Bryan__ joined #gluster
15:04 JoeJulian H__: In libcattr.py I'm not sure, but I'm wondering if you'll get that memory allocation error if the glibc call tries to exceed the size of buf which, for xtime, is expected to be 8 bytes.
15:07 JoeJulian Alpinist: What if you install a package using rpm instead of yum?
15:10 gbr I reboot a couple of VM's, and my errors stopped.  This is just getting weird.
15:11 JoeJulian gbr: which version is that?
15:12 Alpinist I'll try that JoeJulian
15:13 Alpinist What i discovered latest: if i install something with yum and reset the machine, so no clean shutdown, it will boot perfect
15:14 gbr JoeJulian:  3.3.0.  The replicate is running 3.3.1
15:14 Alpinist Than rebooting the machine in a clean way, will trigger the hanging
15:14 H__ JoeJulian: interesting. How would one find out ? It seems a common geo-replication issue as I see this issue pop up on the mailing list a lot.
15:14 nightwalk joined #gluster
15:14 JoeJulian weird
15:15 gbr I haven't had a stable gluster since Nov 1, and it's getting worse.  My stress level is through the roof.
15:16 JoeJulian gbr: did you upgrade ONE of your servers to 3.3.1 on Nov 1?
15:17 nueces joined #gluster
15:17 gbr Nope.  Upgrade sometime around the 15th.  I want to upgrade the other, but this flakiness means my replicate is constantly out of sync.  It says it's in sync now, but can I trust it?
15:18 JoeJulian I would run a heal...full to make sure. I just have no way of knowing if that completes... :/
15:18 gbrand_ joined #gluster
15:19 JoeJulian ... actually, no. I would do the old method of find + stat and check my logs.
15:19 yinyin joined #gluster
15:22 tqrst is it normal that glustershd doesn't die when the glusterd service is stopped? I still see "/usr/sbin/glusterfs -s localhost --volfile-id gluster/glustershd -p /var/lib/glusterd/glustershd/run/glustershd.pid -l /var/log/glusterfs/glustershd.log -S /tmp/075f5667c890e4a30fd7386159588e6d.socket --xlator-option *replicate*.node-uuid=ebf08"
15:23 tqrst (looks like a client, but the volfile-id isn't anything I created, so I'm assuming it's just glustershd)
15:23 wushudoin joined #gluster
15:26 JoeJulian gbr: Any idea what NFS_RAID6_F0 corresponds to?
15:26 JoeJulian tqrst: It is normal.
15:27 robo joined #gluster
15:27 gbr JoeJulian: find/stat ran (very fast, not manu files, just large VM images) and returned no errors in the log
15:27 gbr JoeJulian: I don't understand the question 'corresponds to'
15:28 gbr NFS_RAID6_FO is glusters export of /export/gluster/RAID-6
15:28 JoeJulian Ah, ok.
15:29 JoeJulian So that's the volume name?
15:29 gbr The IP address in the error is one of my XenServers
15:29 gbr yup, volume name.
15:30 JoeJulian "xlatorp->name" isn't very descriptive in the source
15:30 gbr I have the same servers exporting NFS_RAID1_FO, and it's not having these issues.
15:30 gbr As frequently.
15:31 gbr But there is less load on the RAID1, and this seems to be load related.
15:31 JoeJulian Anything in /var/log/messages on 192.168.20.12 maybe?
15:32 JoeJulian ~processes | tqrst
15:32 glusterbot information.
15:32 glusterbot tqrst: the GlusterFS core uses three process names: glusterd (management daemon, one per server); glusterfsd (brick export daemon, one per brick); glusterfs (FUSE client, one per client mount point; also NFS daemon, one per server). There are also two auxiliary processes: gsyncd (for geo-replication) and glustershd (for automatic self-heal). See http://goo.gl/hJBvL for more
15:34 gbr yeah, it's complaining about losing it's NFS server.
15:35 JoeJulian So I would suspect a timeout or a network issue which would go along with your load suspicion.
15:35 gbr How in sync do the gluster servers clocks have to be.  I'm getting clock drift.  I synced them last night, and now they are 11 seconds apart
15:36 JoeJulian It's not hyper critical, but it's important. Can't you ntp them?
15:37 gbr They are ntp'd.  And sorry, they are in sync now, they were out by almost a minute last night.  My ntp server was non-responsive for awhile, due to these issues.
15:38 gbr I'm going to shut down the 3.3.0 server this evening (after I'm sure they are in sync) and do an fsck and gluster 3.3.1 upgrade.
15:39 gbr Two weekends ago, I took them down and did memory tests and drive tests.  Not sure why I didn't upgrade both then.
15:40 gbr Network is plain GigE, and I suspeted network issues, so I swapped out the switch 2 wekends ago.
15:51 ctria joined #gluster
15:51 JoeJulian gbr: Try swapping network cables too. I've seen high load disconnects due to a bad cat6.
15:54 JoeJulian H__: Coming back to your issue... "getfattr -m . -d - e hex" some file on your brick... hopefully one that's already been processed by marker.
15:57 aliguori joined #gluster
15:59 Humble joined #gluster
16:03 daMaestro joined #gluster
16:04 Alpinist JoeJulian, with rpm i've experienced the same problem
16:04 nightwalk joined #gluster
16:05 JoeJulian Alpinist: Can I see an ls of /etc/rc.d/rc3.d please?
16:10 Alpinist JoeJulian, http://goo.gl/YHuh3
16:10 glusterbot Title: Your code. Your site. Use it. - paste.org (at goo.gl)
16:11 Alpinist i don't think it's in the startupscript because when i don't install, upgrade or reinstall something, it boots perfectly
16:12 JoeJulian Right, you said it's haning during netfs, iirc.
16:14 H__ JoeJulian: you mean on a brick of the master right ? bypassing any (local) glusterfs mounts (and without the space between - and e)
16:14 Alpinist yes it is, but only after installing or updating something
16:14 JoeJulian H__: right (and thanks for catching that typo)
16:15 Alpinist JoeJulian,  and if i reset the machine after installing something it boots correct. So i guess there something in the shutdown-process that goed wrong
16:15 H__ JoeJulian: I get no output at all ?!
16:15 gbr joined #gluster
16:15 semiosis :O
16:15 JoeJulian Alpinist: Or something in the shutdown process that's doing something that has to be undone at boot.
16:15 JoeJulian :O
16:16 H__ JoeJulian: ah, i need to be root
16:17 deepakcs joined #gluster
16:18 H__ JoeJulian: here's the output http://dpaste.org/9bGOC/
16:18 glusterbot Title: dpaste.de: Snippet #214374 (at dpaste.org)
16:20 nightwalk joined #gluster
16:25 blendedbychris joined #gluster
16:25 blendedbychris joined #gluster
16:27 Alpinist indeed JoeJulian , but what ... :s
16:27 mohankumar joined #gluster
16:29 chacken1 joined #gluster
16:30 JoeJulian H__: Ok, so if I'm interpreting this correctly there's a trusted.glusterfs.{gfid}.xtime that's more than 8 bytes. I would probably either parse the output of "getfattr -km xtime -R -P -h -e hex $brick" or write a short python script to do the same. You're looking for an xtime that's more than 8 bytes.
16:31 JoeJulian s/km/m/
16:31 glusterbot What JoeJulian meant to say was: H__: Ok, so if I'm interpreting this correctly there's a trusted.glusterfs.{gfid}.xtime that's more than 8 bytes. I would probably either parse the output of "getfattr -m xtime -R -P -h -e hex $brick" or write a short python script to do the same. You're looking for an xtime that's more than 8 bytes.
16:32 nueces joined #gluster
16:35 flakrat joined #gluster
16:36 maccam94 joined #gluster
16:37 JoeJulian Alpinist: How about fstab?
16:38 flakrat Howdy, if we have 6 servers, each with 30TB usable storage, considering future growth, does it make sense for each server to use a single 30TB brick, or break the storage into smaller chunks, say 10 x 3TB
16:38 JoeJulian ... there's got to be something else. All rpm should be doing is installing the package files and updating it's database in /var/lib/rpm.
16:38 JoeJulian flakrat: Yep
16:38 JoeJulian either one of those makes sense.
16:38 flakrat I'm thinking that as we add additional servers, it'll be easier to match a 3TB brick size than a 30TB size
16:39 flakrat or is it now supported to use non uniform brick sizes in a Gluster volume?
16:39 JoeJulian It's not advised to have non uniform brick sizes, no.
16:39 maccam94 anyone know the ramifications of removing trusted.afr.* xattrs on directories? will they get re-created automatically?
16:40 flakrat JoeJulian, thanks I had found a conversation from 2011 regarding that, but nothing newer
16:40 JoeJulian Some use raid 10 under their bricks, some raid 5 or 6, some use lvm and carve out partitions to make bricks for each individual volumes
16:41 ndevos JoeJulian: btw, trusted.glusterfs.{gfid}.xtime is more like trusted.glusterfs.{geo-rep-sessionid}.xtime
16:42 Alpinist services01:/labfs       /labfs  glusterfs        defaults,_netdev       0       0
16:42 JoeJulian maccam94: They may get recreated automatically or they file may just not be visible from the client. Doing a lookup by name will cure that though.
16:42 JoeJulian ndevos: Thanks, I meant uuid but that's a much better representation.
16:42 JoeJulian I was focused on the overflow.
16:42 Alpinist Thank your help JoeJulian, olisch
16:43 Alpinist need to go ;)
16:43 jbrooks joined #gluster
16:44 JoeJulian Yeah, I should probably get going to. Need to get down to the office and do some hardware stuff.
16:44 gbr JoeJulian: Thanks for your help.
16:45 JoeJulian You're welcome. Good luck with that. I hope I helped.
16:47 maccam94 JoeJulian: i'm not sure how that works with directories, since they are on all bricks. i tried ls'ing the directory as well as a file within the directory on the brick, but that did not seem to fix the xattrs
16:49 semiosis maccam94: you'd need to stat the dirs/files through a client mount in order for self-heal to fix the afr xattrs
16:50 semiosis maccam94: ,,(repair)
16:50 glusterbot maccam94: http://goo.gl/uA812
16:50 semiosis that should happen automatically for common causes of inconsistency in glusterfs 3.3+
16:51 maccam94 semiosis: i just stat'd the directory, subdirectory, and file within the subdirectory. xattrs still aren't back. this is on glusterfs 3.2.5
16:51 semiosis maccam94: are you doing that on a brick directly or throuhg a client mount poitn?
16:51 maccam94 stat'ing on the client mountpoint, verifying xattrs on the brick
16:52 semiosis check your client log file, and feel free to pastie.org it too
16:56 nightwalk joined #gluster
16:58 H__ JoeJulian: ehh, ok :) Need a little help on what you said. Are you saying I need to change all xtime attributes to be 8 bytes on all inodes ?
16:59 ras0ir joined #gluster
16:59 ras0ir joined #gluster
17:02 theron joined #gluster
17:05 sjoeboo joined #gluster
17:05 hagarth joined #gluster
17:05 mohankumar joined #gluster
17:10 circut joined #gluster
17:14 nightwalk joined #gluster
17:18 gbrand_ joined #gluster
17:19 maccam94 joined #gluster
17:31 Bullardo joined #gluster
17:33 rastar joined #gluster
17:35 rastar left #gluster
17:35 lh joined #gluster
17:36 Mo_ joined #gluster
17:37 Mo__ joined #gluster
17:39 inodb joined #gluster
17:39 plarsen joined #gluster
17:42 blendedbychris1 joined #gluster
17:45 chirino_m joined #gluster
17:50 tryggvil joined #gluster
17:50 tryggvil_ joined #gluster
17:53 gbrand_ joined #gluster
17:53 inodb joined #gluster
18:02 blendedbychris joined #gluster
18:02 blendedbychris joined #gluster
18:03 inodb_ joined #gluster
18:03 maccam94 joined #gluster
18:08 tqrst JoeJulian: oddly enough, the issues I had yesterday with replace-brick failing on a server went away when I unmounted the volume from that server
18:14 lhawthor_ joined #gluster
18:14 chirino joined #gluster
18:15 lh joined #gluster
18:22 theron joined #gluster
18:22 nightwalk joined #gluster
18:24 tc00per @nfs
18:24 glusterbot tc00per: To mount via nfs, most distros require the options, tcp,vers=3 -- Also portmapper should be running on the server, and the kernel nfs server (nfsd) should be disabled
18:25 tqrst in fact, I've seen a very strong correlation between replace-brick failing and the volume being mounted on that particular server
18:26 rags joined #gluster
18:29 obryan joined #gluster
18:32 quillo joined #gluster
18:33 tqrst is there a reason why the gluster cli doesn't return a non-zero exit code when it encounters an error?
18:33 tqrst oops nevermind, my bad
18:36 bauruine joined #gluster
18:38 chirino joined #gluster
18:42 rastar joined #gluster
18:46 quillo joined #gluster
18:58 nightwalk joined #gluster
18:59 y4m4 joined #gluster
18:59 hattenator joined #gluster
19:00 bennyturns joined #gluster
19:05 gbr joined #gluster
19:13 rags_ joined #gluster
19:16 dalekurt joined #gluster
19:17 nightwalk joined #gluster
19:17 chirino joined #gluster
19:27 mooperd joined #gluster
19:27 jbrooks joined #gluster
19:32 chirino joined #gluster
19:33 andreask joined #gluster
19:35 nightwalk joined #gluster
19:38 elyograg gbr: ntp is *supposed* to make servers able to run with proper synchronization even when the time server goes away.  That's what the drift calculation is all about.
19:39 nightwalk joined #gluster
19:39 gbr elyograg: maybe chrony has an issue with that.
19:40 elyograg gbr: you might need to add a "server 127.127.1.0" line to your ntp.conf to make that happen, though.  that tells it to use itself as a time source.
19:41 elyograg gbr: never used chrony.  always used the ntp implementation available from the distro packagers.
19:48 gbrand_ joined #gluster
19:52 nightwalk joined #gluster
19:56 chirino joined #gluster
19:56 tqrst joined #gluster
19:56 tqrst joined #gluster
20:11 y4m4 joined #gluster
20:12 nightwalk joined #gluster
20:15 glusterbot New news from newglusterbugs: [Bug 881997] gluster volume geo-replication config checkpoint now works but returns command failed <http://goo.gl/OiUKK>
20:18 nueces joined #gluster
20:18 inodb joined #gluster
20:24 plarsen joined #gluster
20:26 nightwalk joined #gluster
20:27 rags joined #gluster
20:29 lh joined #gluster
20:29 lh joined #gluster
20:42 JoeJulian Is John Read in here?
20:44 tryggvil_ joined #gluster
20:44 tryggvil joined #gluster
20:45 JoeJulian H__: What I'm curious about is whether there's an xtime that exceeds 8 bytes. There shouldn't be. If there is, that's where the problem is. Not sure how to fix it yet if there is, but one thing at a time.
20:46 purpleidea joined #gluster
20:49 nueces joined #gluster
20:54 nueces_ joined #gluster
20:57 _NiC joined #gluster
20:57 saz_ joined #gluster
21:06 H__ JoeJulian: ok. I just picked a file, for all i know all my files have >8bytes xtime attributes. /me confused
21:07 JoeJulian H__: Well the one you showed me does have an 8 byte xtime. You're looking for one that's longer than 0xNNNNNNNN
21:11 mooperd joined #gluster
21:11 nightwalk joined #gluster
21:19 H__ sorry, i misunderstood. So the postulate is that an >8bytes xtime attr would cause the memory error I pasted earlier. Right ?
21:20 AK6L joined #gluster
21:21 AK6L hi folks, have a question about Gluster in EC2
21:21 __Bryan__ joined #gluster
21:21 AK6L trying to set up a cluster against Elastic IPs; having some trouble
21:21 AK6L i have three nodes, each with their own EIP; from the '01' node (dev-gluster-01) i do 'gluster peer probe dev-gluster-02' and '... dev-gluster-03'
21:21 AK6L it all works
21:22 JoeJulian H__: Right!
21:22 AK6L but if i do a 'gluster peer status' on the other nodes, i see the EIP of dev-gluster-01, not the hostname
21:22 JoeJulian ~hostnames | AK6L
21:22 glusterbot AK6L: Hostnames can be used instead of IPs for server (peer) addresses. To update an existing peer's address from IP to hostname, just probe it by name from any other peer. When creating a new pool, probe all other servers by name from the first, then probe the first by name from just one of the others.
21:22 AK6L which means i can't do a 'gluster volume create' ...
21:22 AK6L ok
21:22 AK6L i'll try that, thanks
21:28 tqrst is it normal for a folder to have no trusted.afr.* entries at all on certain bricks? Or does that also count as the value being 0?
21:32 nightwalk joined #gluster
21:34 pdurbin joined #gluster
21:35 pdurbin jdarcy: sorry i had to run! you were just getting into the code! http://irclog.greptilian.com/sourcefu/2012-12-04
21:35 glusterbot <http://goo.gl/5BUrX> (at irclog.greptilian.com)
21:35 pdurbin kkeithley1: nice to put a name to a face
21:36 pdurbin johnmark: a pleasure, as always
21:38 AK6L hmm, that didn't work
21:38 pdurbin JoeJulian and semiosis: you're awesome: http://irclog.perlgeek.de/cr​imsonfu/2012-12-04#i_6212671
21:38 glusterbot <http://goo.gl/Z0jby> (at irclog.perlgeek.de)
21:39 AK6L here's what i'm still getting: http://pastebin.com/MDDARCKV
21:39 glusterbot Please use http://fpaste.org or http://dpaste.org . pb has too many ads. Say @paste in channel for info about paste utils.
21:40 AK6L fine: http://fpaste.org/riOc/
21:40 glusterbot Title: Viewing Paste #257502 (at fpaste.org)
21:40 rbennacer joined #gluster
21:40 rbennacer stopbit, hi
21:40 rbennacer :)
21:41 stopbit hello, rbennacer
21:41 AK6L anyone have any thoughts on why i can't 'gluster volume create' with hostnames, even though i see all the proper hostnames in 'gluster peer status'?
21:41 AK6L and the hostnames are listed in /etc/hosts
21:41 AK6L this is in EC2, hostnames refer to EIPs
21:43 AK6L guess i have to use the public DNS for the EIPs.  annoying.
21:45 semiosis pdurbin: rock on
21:46 semiosis AK6L: you can
21:46 mjrosenb this is awkward, I just started getting lots of duplicat (and empty) files
21:46 semiosis you probably need to map each machine's own hostname to 127.0.0.1 in its /etc/hosts file
21:46 AK6L hmm, ok, i'll try that
21:49 Luis_Silva joined #gluster
21:50 AK6L semiosis: that did it, thanks
21:50 semiosis yw
21:50 Luis_Silva left #gluster
21:51 nightwalk joined #gluster
21:51 clutchk joined #gluster
21:53 rbennacer i can't start glusterfd for an unknown reason, it was working just fine but after a reboot it is crashing after the start. can anyone make sens out of the logs? http://pastebin.com/Af7sWGvL
21:53 glusterbot Please use http://fpaste.org or http://dpaste.org . pb has too many ads. Say @paste in channel for info about paste utils.
21:56 rbennacer here is the logfile after using fpaste   heheheh. http://fpaste.org/sejv/
21:56 glusterbot Title: Viewing Paste #257509 (at fpaste.org)
21:57 mjrosenb so is it reasonable that I have a directory with every file on existing on both bricks?
21:57 mjrosenb but complementary sets are empty on each brick?
22:09 H__ Just found out that rebalance did not spread data evenly over servers A+B after adding B, what can I do to make the setup HA now ?
22:10 nightwalk joined #gluster
22:18 bennyturns joined #gluster
22:29 maccam94 joined #gluster
22:36 nightwalk joined #gluster
22:39 77CABV6GJ joined #gluster
22:43 DaveS joined #gluster
22:47 nightwalk joined #gluster
22:56 theron joined #gluster
23:01 TSM2 joined #gluster
23:04 gbr joined #gluster
23:06 noob2 joined #gluster
23:06 noob2 in the next version of gluster is the fuse client going to be deprecated?
23:06 rbennacer left #gluster
23:06 nightwalk joined #gluster
23:07 semiosis noob2: where'd you get that idea?
23:07 noob2 lol
23:07 noob2 i heard a rumor
23:07 noob2 here's what i'm wondering
23:08 lh joined #gluster
23:08 lh joined #gluster
23:08 noob2 when i  start to scale out with the  gluster and the fuse client i think i'm going to run into a network connection limit
23:08 noob2 because each client connects to each brick
23:08 noob2 i'm sure others have encountered this as well.  hence i thought maybe the fuse client wouldn't be the best direction in the future?
23:10 noob2 i apologize if i'm way off base
23:12 noob2 semiosis: i did have another question. how was gluster able to get mountall to recognize the gluster mount type in fstab?  i was thinking of adding a ceph type to ubuntu and it's a similar problem
23:12 noob2 and i guess by gluster i mean redhat
23:16 kkeithley1 pdurbin: likewise
23:17 mooperd joined #gluster
23:18 nightwalk joined #gluster
23:24 nueces joined #gluster
23:26 noob2 semiosis: a crazy thought experiment.  lets say my manager comes in one day with a crazy idea that involves setting up gluster on 1,000 nodes with 12 disks in each.  With the fuse client i'm looking at 12,000 network connections right?
23:26 noob2 with the fuse client i'm thinking i could only connect about 5 clients before the server runs out of possible network numbers
23:27 a2 depends how you manage the 12 disks on the server.. if you lvm them into a single backend filesystem, then only 1000 connections from each fuse client
23:27 noob2 right
23:27 noob2 but still you see where i'm going with this
23:27 noob2 how would you scale out wide like that without bottlenecking one server by using nfs
23:27 a2 yep, 64k ports is a limitation
23:28 noob2 yeah that's what i was thinking
23:28 noob2 maybe if you ctdb'd with nfs you'd overcome that?
23:28 a2 not sure if it is a limitation for 1000 nodes, but if you create too many volumes then it can be
23:28 noob2 ok
23:29 noob2 i was thinking just one massive volume for conversation sake
23:29 noob2 you'd have about 12 petabytes i think if you used 3TB drives
23:29 noob2 maybe my math is off, that sounds low
23:31 noob2 google says that is right.  petabytes are a lot larger than i thought
23:31 nightwalk joined #gluster
23:35 semiosis noob2: i dont think you could avoid that by ditching the fuse client... gluster's nfs daemon would need to maintain those connections just the same
23:36 semiosis you'd need some new way of managing tcp connection state that doesn't rely on one local port per connection
23:36 semiosis which seems possible in theory but idk if kernel tcpip stacks can do that
23:37 Technicool joined #gluster
23:38 quillo joined #gluster
23:39 blendedbychris joined #gluster
23:39 blendedbychris joined #gluster
23:41 noob2 i gotcha
23:41 noob2 that makes sense
23:42 noob2 i suppose you could cut it up into different volumes or merge disks like a2 says
23:46 nightwalk joined #gluster
23:49 noob2 so i guess that puts on cap on how big i can physically go even though the software supports going way larger.  into the brontobytes :)
23:55 semiosis afaik people usually run out of money before they run out of tcp ports
23:56 noob2 i agree.  that'd run about 10 million i think to setup

| Channels | #gluster index | Today | | Search | Google Search | Plain-Text | summary