Perl 6 - the future is here, just unevenly distributed

IRC log for #gluster, 2015-10-21

| Channels | #gluster index | Today | | Search | Google Search | Plain-Text | summary

All times shown according to UTC.

Time Nick Message
00:00 zhangjn joined #gluster
00:13 EinstCrazy joined #gluster
00:45 atalur joined #gluster
00:47 halloo joined #gluster
00:49 PaulCuzner joined #gluster
00:50 aravindavk joined #gluster
00:55 EinstCrazy joined #gluster
00:57 vimal joined #gluster
01:01 zhangjn joined #gluster
01:07 Trefex joined #gluster
01:09 Trefex joined #gluster
01:15 kbyrne joined #gluster
01:22 zhangjn joined #gluster
01:30 cholcombe joined #gluster
01:31 Lee1092 joined #gluster
01:32 zhangjn joined #gluster
01:34 theron joined #gluster
01:36 ChrisHolcombe joined #gluster
01:40 halloo joined #gluster
01:41 shyam joined #gluster
01:42 a_ta joined #gluster
01:50 18WAAYLAU joined #gluster
02:02 haomaiwa_ joined #gluster
02:02 calavera joined #gluster
02:06 monotek1 joined #gluster
02:09 PaulCuzner joined #gluster
02:37 halloo joined #gluster
02:42 nangthang joined #gluster
02:50 nangthang joined #gluster
02:53 maveric_amitc_ joined #gluster
02:55 Peppaq joined #gluster
03:02 haomaiwang joined #gluster
03:09 kotreshhr joined #gluster
03:13 bennyturns joined #gluster
03:14 bennyturns joined #gluster
03:23 [7] joined #gluster
03:27 kotreshhr left #gluster
03:27 overclk joined #gluster
03:34 nbalacha joined #gluster
03:40 stickyboy joined #gluster
03:40 stickyboy joined #gluster
03:47 shubhendu joined #gluster
04:01 haomaiwa_ joined #gluster
04:03 ramteid joined #gluster
04:10 night joined #gluster
04:12 haomaiwang joined #gluster
04:12 zhangjn joined #gluster
04:21 gem joined #gluster
04:23 zhangjn joined #gluster
04:24 gem_ joined #gluster
04:26 vmallika joined #gluster
04:27 rafi joined #gluster
04:28 neha_ joined #gluster
04:36 adamaN joined #gluster
04:43 a_ta joined #gluster
04:43 deepakcs joined #gluster
04:45 beeradb joined #gluster
04:48 sakshi joined #gluster
04:49 RameshN joined #gluster
04:51 sghatty_ joined #gluster
04:54 Saravana_ joined #gluster
05:01 haomaiwang joined #gluster
05:02 hagarth joined #gluster
05:03 karnan joined #gluster
05:07 vmallika joined #gluster
05:08 ppai joined #gluster
05:15 kdhananjay joined #gluster
05:22 kovshenin joined #gluster
05:30 Bhaskarakiran joined #gluster
05:33 gem1 joined #gluster
05:35 poornimag joined #gluster
05:46 atalur joined #gluster
05:46 hagarth joined #gluster
05:47 hgowtham joined #gluster
05:47 jiffin joined #gluster
05:54 nangthang joined #gluster
05:58 kanagaraj joined #gluster
05:59 atalur_ joined #gluster
06:01 ashiq joined #gluster
06:01 raghu joined #gluster
06:02 haomaiwa_ joined #gluster
06:03 skoduri joined #gluster
06:03 ramky joined #gluster
06:06 hagarth joined #gluster
06:08 rjoseph joined #gluster
06:09 nishanth joined #gluster
06:15 xavih joined #gluster
06:21 atalur_ joined #gluster
06:22 itisravi joined #gluster
06:26 jtux joined #gluster
06:36 jtux joined #gluster
06:37 B21956 joined #gluster
06:46 vmallika joined #gluster
06:47 Philambdo joined #gluster
06:53 atalur_ joined #gluster
07:01 fsimonce joined #gluster
07:01 haomaiwa_ joined #gluster
07:08 [Enrico] joined #gluster
07:14 LebedevRI joined #gluster
07:15 Saravana_ joined #gluster
07:18 ivan_rossi joined #gluster
07:19 zhangjn joined #gluster
07:20 zhangjn joined #gluster
07:37 a_ta joined #gluster
07:37 Sunghost joined #gluster
07:38 gem joined #gluster
07:39 atalur_ joined #gluster
07:41 Sunghost Hello, i got still the problem that files not on mountpoint but on brick on distribted glusterfs. i also miss files if i copy them on mount.
07:43 EinstCrazy joined #gluster
07:44 deniszh joined #gluster
07:45 ctria joined #gluster
07:46 julim joined #gluster
07:47 gem joined #gluster
07:48 DV__ joined #gluster
07:49 zhangjn joined #gluster
08:00 anti[Enrico] joined #gluster
08:02 haomaiwa_ joined #gluster
08:02 EinstCrazy joined #gluster
08:02 yoavzz joined #gluster
08:05 yoavzz Hi, I’m having error in production that I can’t fix. We had a 2 node cluster with 4 volumes, replica 2, one brick on each server. nnfs02 / nnfs03 it’s their hostnames. nnfs02 failed and we can’t start the server. I managed to remove it from some volumes but in two volumes it wasn’t possible. On two volumes I managed to remove the old bricks and create new bricks on the replacement server (nnfs01).
08:07 yoavzz I wanted to restart the gluster service on nnfs03 to clear the ghost data about nnfs02 and now glusterd won’t start. It says a lot of things, but the main issue is that it can’t find the dead node.
08:09 yoavzz debug log: https://gist.github.com/anonymous/d8c412bda8754b4b67a1
08:09 glusterbot Title: gist:d8c412bda8754b4b67a1 · GitHub (at gist.github.com)
08:17 thoht_ during heal, is it a normal behavior that the file in heal progress is locked, i mean i cannot read it anymore
08:28 dusmant joined #gluster
08:28 mlhamburg1 joined #gluster
08:32 anti[Enrico] joined #gluster
08:33 halloo joined #gluster
08:35 jwd joined #gluster
08:39 deniszh joined #gluster
08:41 DV__ joined #gluster
08:43 DV joined #gluster
08:46 tanslep joined #gluster
08:48 tanslep Hi
08:48 glusterbot tanslep: Despite the fact that friendly greetings are nice, please ask your question. Carefully identify your problem in such a way that when a volunteer has a few minutes, they can offer you a potential solution. These are volunteers, so be patient. Answers may come in a few minutes, or may take hours. If you're still in the channel, someone will eventually offer an answer.
08:48 SOLDIERz joined #gluster
08:49 tanslep Is it safe to delete a volume? Will that destroy data? I created a distribute volume but want to change to a replicated volume
08:55 jiffin tanslep: u can delete the volume, data will be still available in the disk(brick)
08:56 tanslep Thanks jiffin
08:57 jiffin tanslep: i think u can convert a distributed volume into replica using add brick , but i am not sure about exact steps
08:57 jiffin itisravi: any clues??
08:57 Sunghost hi i have a problem - data copied on mount of volume is not at all shown, but on brick
08:58 itisravi tanslep: If you just want to convert a distribute vol to a distribute replicate, you can do so using the add-brick command.
09:00 tanslep Itisravi:do you know what switched to use?
09:01 jiffin Sunghost: may be ur brick is offline mode, you might need to start the volume again
09:02 haomaiwa_ joined #gluster
09:03 tanslep I already have two bricks. So would it be best to remove a brick and then add it again?
09:04 Sunghost hi jiffin - i have a to brick distributed - each with raid6 - on brick2 i lost 3 disks, 1 of them had lots of bad blocks, however i could recover the raid
09:04 Sunghost but lost 10tb data
09:04 Sunghost now both bricks are online and all is running except the lost of 10tb
09:04 Sunghost 2 problems
09:04 Sunghost 1 - i see some files not on mount but on brick1 or brick2
09:05 Sunghost 2 - if i copy new files over mount, i see some files other not, but on brick1 or brick2
09:08 kotreshhr joined #gluster
09:09 taolei joined #gluster
09:09 jiffin Sunghost: i have never heard about this find of issues, can u do two things 1.) give output gluster v status  2.) try to perform a explicit lookup on that path over the mount (ls <path of file in the mount>) and check what it says
09:10 Sunghost last one - no files or folder
09:11 Sunghost status shows all online
09:11 Sunghost tried this
09:12 Sunghost if no solution can be found i whould create a new volume and copy all from mount vol1 to vol2 and afterwards all from brick dir into mount - could this work?
09:12 yoavzz Hi, If anyone here can try to assist with my gluster issues, it’ll be great - http://serverfault.com/questions/730502/glusterd-wont-start-after-node-crash
09:12 glusterbot Title: linux - Glusterd won't start after node crash - Server Fault (at serverfault.com)
09:12 yoavzz I wrote in here earlier but had no response
09:13 taolei Is it possible to specify interface(s) which glusterd and glusterfsd will listen on?
09:15 jiffin Sunghost: that ideally work , can u provide the client logs when perform the explicit lookup and also try this on missing files from the mount at the brick getfattr -d -m . -e hex <file path in brick>
09:16 thoht_ is it really better to use direct-io-mode=disable on mount option for better perfs on clients ?
09:18 harish joined #gluster
09:19 hagarth yoavz: looks like nnfs02 is not resolvable on the node where glusterd did not restart
09:21 spalai joined #gluster
09:23 jiffin taolei: are u talking about interfaces like tcp, rdma or  just nic like 1Gb, 10Gb etc ??
09:24 itisravi tanslep:  gluster vol add-brick volname replica 2 newbrick1 newbrick2 ..etc
09:24 taolei jiffin: Hi, I mean the IP address, not NIC.
09:25 tanslep Itisravi: the bricks are already in a volume. Should I remove the bricks and then add them again?
09:26 taolei jiffin: I want gluster daemons to listen on certain ip addresses, not 0.0.0.0
09:27 jiffin taolei: sorry not sure about that
09:28 yoavzz joined #gluster
09:28 itisravi tanslep: what is the output of gluster volume info volname?
09:31 tanslep Type:Distribute. Status:started. Number of bricks:2. Transport-type:tcp. Brick1:Zeus:/data. Brick2:heracles:/data. Options reconfigured: features.quota:on
09:32 Bhaskarakiran_ joined #gluster
09:35 tanslep Itisravi:tanslep Type:Distribute. Status:started. Number of bricks:2. Transport-type:tcp. Brick1:Zeus:/data. Brick2:heracles:/data. Options reconfigured: features.quota:on
09:36 Sunghost ok i test it again, actuall i copy some files over nfs mount - i see on both bricks the new directory and on brick2 some files
09:36 Sunghost on another client with nfs mount i see not the folder
09:36 rjoseph joined #gluster
09:38 itisravi tanslep: okay, so you can add 2 more bricks and convert it into a distrivbuted-replicate volume.
09:38 itisravi tanslep: is that what you want?
09:38 tanslep Itisravi no I want to change the current distribute volume with those two bricks to a replicate
09:39 itisravi tanslep: ah! that would not be possible.
09:39 RayTrace_ joined #gluster
09:39 stickyboy joined #gluster
09:40 tanslep Itisravi. Ok thanks. Can I remove the bricks and then recreate them?
09:41 itisravi tanslep:  you cannot use existing bricks to create a new volume.
09:42 tanslep Itisravi oh dear! Can I delete the volume and start again?
09:42 kanagaraj_ joined #gluster
09:42 itisravi tanslep: do you have another set of bricks that you can use to create a new replicate volume?
09:43 Sunghost notice - i extract files into a new folder and cant see the folder with ls, but can go into with cd, but cant see files
09:44 tanslep Itisravi: no I don't. I only have those two bricks. I just want to start again!
09:44 Sunghost files are on problem brick2 perhaps thats the reason
09:45 arcolife joined #gluster
09:45 tomatto joined #gluster
09:45 itisravi tanslep: ok, and I assume you want the data in the bricks too? Or do you not care about it?
09:45 Sunghost should i run this from client ? getfattr -d -m . -e hex New\ folder
09:46 itisravi tanslep: If you do, the only way is to copy the data elsewhere, delete the volume and create a new one and copy the data into the new volume.
09:48 tanslep Itisravi: I do need the data but there is about 20tb!  Oh well
09:49 itisravi tanslep: tough luck :-(
09:50 tanslep Itisravi: thanks for your help.
09:51 itisravi tanslep: you're welcome.
09:51 tanslep Quit
09:51 tanslep left #gluster
09:58 sage joined #gluster
09:58 Sunghost shout i run "getfattr -d -m . -e hex New\ folder" from client or direct on brick2?
10:02 haomaiwa_ joined #gluster
10:05 suliba joined #gluster
10:10 Sunghost getfattr from brick2 on which the files laying
10:10 Sunghost trusted.gfid=0x3bbda26a3fc84c0288f6da40ea056b67
10:10 Sunghost trusted.glusterfs.dht=0x000000010000000000000000a92dcf2f
10:11 Sunghost from brick1: trusted.gfid=0x3bbda26a3fc84c0288f6da40ea056b67
10:11 Sunghost trusted.glusterfs.dht=0x0000000100000000a92dcf30ffffffff
10:11 Sunghost for new folder
10:14 Bhaskarakiran joined #gluster
10:21 Sunghost itisravi??
10:25 yoavzz joined #gluster
10:27 jiffin Sunghost: from ur messages , this is what i understand , from if u mount with one of the nfsservers everything is fine , but from another it has some missing file issue etc.
10:27 monotek1 joined #gluster
10:27 Sunghost infrastructure 1 client 2 bricks with vol1 in distributed
10:28 jiffin k
10:28 Sunghost client mount via nfs on brick1
10:28 Sunghost brick2 was crashed
10:28 Sunghost copy over nfs from client1 -> files on brick2 but not on nfs mount
10:28 Sunghost icant see the folder
10:29 jiffin what about nfsserver in brick2??
10:30 Sunghost how do you mean that?
10:30 bluenemo joined #gluster
10:31 jiffin can u please give me the output of volume status command from the node in brick1 and the node in the brick2??
10:32 Sunghost testet nfs mount on brick2 with brick1 and brick2 but no folder and files
10:32 bfoster joined #gluster
10:32 Sunghost gluster volume status
10:32 Sunghost Status of volume: vol2
10:32 Sunghost Gluster processPortOnlinePid
10:32 Sunghost ------------------------------------------------------------------------------
10:32 glusterbot Sunghost: ----------------------------------------------------------------------------'s karma is now -8
10:32 Sunghost Brick clusternode01:/media/node01/vol249155Y1008
10:32 Sunghost Brick clusternode02:/media/node02/vol249154Y2593
10:32 Sunghost NFS Server on localhost2049Y2588
10:32 Sunghost NFS Server on 192.168.0.412049Y1013
10:32 Sunghost
10:32 Sunghost Task Status of Volume vol2
10:33 Sunghost ------------------------------------------------------------------------------
10:33 glusterbot Sunghost: ----------------------------------------------------------------------------'s karma is now -9
10:33 Sunghost There are no active volume tasks
10:33 Sunghost same on brick1
10:33 Sunghost seems ok
10:33 jiffin1 joined #gluster
10:34 neha__ joined #gluster
10:34 Sunghost one difference is that the nfs server on brick 2 is written with ip-adress and on brick1 with hostname
10:34 EinstCrazy joined #gluster
10:35 Trefex joined #gluster
10:35 jiffin1 can u please provide me "output"??
10:36 Trefex JoeJulian: my problem with oVirt was indeed due to different versions it seems
10:36 Sunghost same as above but nfs server on shows hostname instead of ip-adress
10:36 a_ta joined #gluster
10:37 Sunghost but hostname and ip-adress are resolveable
10:37 kovshenin joined #gluster
10:38 jiffin Sunghost: can u please paste the output in fpaste/pastbin and give me the link
10:39 kanagaraj joined #gluster
10:41 kanagaraj_ joined #gluster
10:43 firemanxbr joined #gluster
10:44 kanagaraj__ joined #gluster
10:48 kanagaraj joined #gluster
10:51 Sunghost sure -> http://pastebin.com/iqxBzcA3
10:51 glusterbot Please use http://fpaste.org or http://paste.ubuntu.com/ . pb has too many ads. Say @paste in channel for info about paste utils.
10:55 Sunghost in brick log of brick2 i see
10:55 Sunghost [posix.c:4758:posix_fstat] 0-vol2-posix: pfd is NULL, fd=0x55fa861a5808
10:55 Sunghost [2015-10-21 10:32:22.759176] I [server-rpc-fops.c:1235:server_fstat_cbk] 0-vol2-server: 636: FSTAT -2 (b77c9749-d6c5-47df-a386-fc1e1c6c4dd4) ==> (Operation not permitted)
10:55 Sunghost perhaps a problem in populating files to glusterfs?!
10:57 dblack joined #gluster
11:00 kanagaraj_ joined #gluster
11:00 Sunghost shout i run fix-layout after the crash?
11:02 jiffin Sunghost: i am not sure that will help , just try perform  fix-layout and followed by a rebalance
11:02 Sunghost could i lost data? or shout i simple create on same bricks new volume und move data to it? first from mountpoint and later from bricks ?
11:03 Sunghost brick2 etc log: Unable to get transaction opinfo for transaction ID : e6390718-d86a-46ad-adea-51b327835041
11:03 neha__ joined #gluster
11:04 jiffin Sunghost: u should lost the data, but i am not sure whether it will solve ur problem or not
11:04 jiffin s/should/should not.
11:04 Sunghost ?! not right?
11:05 Trefex jiffin: most intense sed line ever
11:05 jiffin u should not lost data
11:05 Sunghost ;)
11:06 Sunghost ok i think i can simple test it -> create new vol and copy files from client via nfs to newvol - if i see it on mount its ok, when not there might be another problem
11:06 jiffin Sunghost: sure
11:07 Sunghost sadly that here is no other expert for helping
11:07 glusterbot Sunghost: Please don't naked ping. http://blogs.gnome.org/markmc/2014/02/20/naked-pings/
11:13 kkeithley1 joined #gluster
11:13 yoavzz joined #gluster
11:16 kanagaraj__ joined #gluster
11:16 hagarth yoavzz: looks like nnfs02 is not resolvable on the node where glusterd did not restart
11:16 yoavzz hagarth: it’s in the hosts file. but the machine’s dead.
11:18 hagarth yoavzz: hmm, atleast that was the reason I noticed in your glusterd's log file
11:18 yoavzz Also, I checked the config files under /var/lib/glusterfs - it’s a total mix up with the old dead server. I wish there was a way to just rebuild everything without losing data
11:19 hagarth yoavzz: do you mean /var/lib/glusterd and what's the mix up?
11:19 yoavzz hagarth: the volumes that are having problems still things it’s a replica 3 and not replica 2 (like it should). the bricks on the dead server are still there.
11:20 yoavzz hagarth: It looks like there’s a lot of junk there.
11:20 hagarth yoavzz: do you know how that happened?
11:21 yoavzz hagarth: nnfs02 died, we wanted to add nnfs01 to replace nnfs02. Instead of replace-brick I used remove-brick & add-brick. I couldn’t do it for all volumes
11:22 yoavzz Nothing went smooth :(
11:23 hagarth yoavzz: did nnfs01 contain a single brick in a replica set for all volumes?
11:25 kanagaraj_ joined #gluster
11:27 tomatto joined #gluster
11:29 kanagaraj__ joined #gluster
11:31 yoavzz hagarth: I’m not sure I understand your question. Every brick is a replica of a volume.
11:33 yoavzz s/replica/replica brick/
11:33 glusterbot What yoavzz meant to say was: hagarth: I’m not sure I understand your question. Every brick is a replica brick of a volume.
11:33 hagarth yoavzz: did you have more than one brick on nnfs02 part of the same replicated set? I assume not.
11:34 yoavzz hagarth: I think not.
11:34 ramky joined #gluster
11:38 kanagaraj joined #gluster
11:39 hagarth yoavzz: is nnfs02 part of the trusted storage pool now?
11:39 yoavzz hagarth: nnfs02 is dead. I ran “peer detach”.
11:40 hagarth yoavzz: then you should not have nnfs02 in any of your volume configurations. If it does, glusterd will not restart properly.
11:41 yoavzz hagarth: Can I safely edit the configuration files? I’m afraid that it’ll get broken in a way that can’t be fixed :(
11:41 yoavzz hagarth: At the moment, the situation is pretty bad, glusterd is down. But I can still access the data
11:42 EinstCrazy joined #gluster
11:42 hagarth yoavzz: do you have similar brick paths on nnfs01 as nnfs02?
11:43 yoavzz hagarth: on a working volume, which I managed to get replicated across nnfs01 & nnfs03 I see this in the configuration file: brick-0=nnfs03:-data-gluster-source
11:43 yoavzz brick-1=nnfs01:-data-gluster-source
11:44 hagarth yoavzz: take a backup of /var/lib/glusterd; replace nnfs02 with nnfs01 and then restart glusterd everywhere. do you have only 2 nodes in the trusted storage pool?
11:44 morse joined #gluster
11:44 yoavzz hagarth: this is a volume that I managed to set correctly, removed nnfs02 and added nnfs01. In the two volumes that I can’t fix there are still references to the late nnfs02.
11:45 kkeithley1 joined #gluster
11:46 hagarth yoavzz: you should be able to replace all occurences of the late nnfs02 with nnfs01 in /var/lib/glusterd. (take a backup of /var/lib/glusterd on all nodes nevertheless).
11:47 yoavzz hagarth: is there a possiblity that I’ll lost the data somehow?
11:47 hagarth yoavzz: does nnfs03 contain the right data as of now?
11:48 yoavzz hagarth: Yes. For two volumes out of four.
11:48 hagarth yoavzz: what about the other two volumes?
11:49 jiffin1 joined #gluster
11:49 yoavzz hagarth: I have four volumes. avatar source covers ebooks. avatar & source works fine with nnfs01 & 03. covers and ebooks doesn’t work. I tried to remove nnfs02 before glusterd died on me.
11:51 hagarth yoavzz: ok, if IIUC covers and ebooks still have references to nnfs02?
11:51 yoavzz hagarth: correct.
11:51 hagarth yoavzz: are clients accessing covers and ebooks volumes at the moment?
11:52 yoavzz hagarth: yes. using nnfs03 as the endpoint.
11:55 hagarth yoavzz: would it be possible to schedule a downtime?
11:56 maveric_amitc_ joined #gluster
11:58 yoavzz hagarth: Maybe, but a short one. 10-30 minutes. If you suggest copy the data and rebuild the array, it’ll take days - it’s a 12TB storage on MS Azure, so it’s fairly slow.
11:58 hagarth yoavzz: that should be good enough. in the downtime I would do this:
11:58 hagarth 1. kill all gluster processes (glusterfs, glusterfsd, glusterd).
11:59 hagarth 2. cp /var/lib/glusterd on both nodes and have a backup.
11:59 kkeithley joined #gluster
11:59 hagarth 3. search and replace nnfs02 with nnfs01/ in /var/lib/glusterd/ on both nodes.
11:59 hagarth 4. restart glusterd
11:59 hagarth that should be it.
12:00 hagarth 5. Ensure that everything looks fine with gluster volume status
12:00 jdarcy joined #gluster
12:00 hagarth self-healing should take care of re-synchronizing data back to nnfs01
12:02 tomatto joined #gluster
12:04 lalatenduM joined #gluster
12:07 yoavzz hagarth: Thanks for the detailed explanation. One question, on the two volumes that aren’t working I added nnfs01 but nnfs02 wasn’t removed. Can I just remove it from there and adjust the replica count accordingly?
12:09 itisravi joined #gluster
12:10 atinm joined #gluster
12:12 hagarth yoavzz: you should be able to do that .. can you fpaste /var/lib/glusterd/<volname>/info from one of the volumes?
12:14 yoavzz hagarth: http://paste.fedoraproject.org/281864/45429655
12:14 glusterbot Title: #281864 Fedora Project Pastebin (at paste.fedoraproject.org)
12:14 RayTrace_ joined #gluster
12:14 yoavzz hagarth: I assume that replica and count values should be changed, although I’m not familiar with “count”’s meaning or usage.
12:15 hagarth yoavzz: yes, all of them should reflect 2
12:16 itisravi joined #gluster
12:16 yoavzz hagarth: so removing the nnfs02 line, change cound and replica to 2. Should I also remove the file from: bricks/nnfs02:-data-gluster-ebooks
12:17 lpabon joined #gluster
12:22 Slashman joined #gluster
12:26 hagarth yoavzz: yes, do remove lines containing nnfs02
12:26 hagarth yoavzz: make sure that contents of this file are the same on both alive nodes
12:26 yoavzz hagarth: and remove the brick files?
12:27 hagarth yoavzz: add brick-0=nnfs01...
12:27 hagarth there should be no brick-2 line in this file
12:28 yoavzz hagarth: ok, I’ll check it out. Going to the business guys to ask for downtime. THANKS A LOT!
12:28 hagarth yoavzz: good luck!
12:30 Trefex joined #gluster
12:32 ppai joined #gluster
12:34 rwheeler joined #gluster
12:38 shyam joined #gluster
12:44 mlhamburg hagarth: We've talked on friday about a problem with ping-timeouts, where your first analysis said that a socket's private structure has become invalid. I filed a bug about this https://bugzilla.redhat.com/show_bug.cgi?id=1272940
12:44 glusterbot Bug 1272940: high, unspecified, ---, bugs, NEW , Shd can't reconnect after ping-timeout (error in polling loop; invalid argument: this->private)
12:47 skoduri joined #gluster
12:49 unclemarc joined #gluster
12:51 jbrooks joined #gluster
13:04 hagarth mlhamburg: thanks, will look into this one.
13:06 mhulsman joined #gluster
13:13 kotreshhr left #gluster
13:17 yoavzz joined #gluster
13:21 mlhamburg hagarth: thanks.
13:24 hagarth joined #gluster
13:30 hamiller joined #gluster
13:31 dgandhi joined #gluster
13:35 bennyturns joined #gluster
13:36 bennyturns joined #gluster
13:41 spalai left #gluster
13:45 rafi joined #gluster
13:47 klaxa|work joined #gluster
13:47 maserati joined #gluster
13:50 Humble joined #gluster
13:56 unicky joined #gluster
13:58 mpietersen joined #gluster
13:59 gem joined #gluster
14:00 a_ta joined #gluster
14:04 cuqa joined #gluster
14:05 cuqa joined #gluster
14:05 rafi joined #gluster
14:13 mpieters_ joined #gluster
14:13 sadbox joined #gluster
14:14 mpieters_ joined #gluster
14:16 shubhendu joined #gluster
14:16 mpieters_ joined #gluster
14:18 mpietersen joined #gluster
14:23 ira joined #gluster
14:23 kovshenin joined #gluster
14:32 wehde joined #gluster
14:44 brian_ when I mount my glusterfs to my volume that is running, and try to create a file in it - it says it's a read only file system
14:46 brian_ https://paste.fedoraproject.org/281949/14454387/
14:46 Slashman joined #gluster
14:46 glusterbot Title: #281949 Fedora Project Pastebin (at paste.fedoraproject.org)
14:55 mhulsman joined #gluster
14:55 maserati @paste
14:55 glusterbot maserati: For a simple way to paste output, install netcat (if it's not already) and pipe your output like: | nc termbin.com 9999
14:57 maserati Am I doing this right? http://fpaste.org/281952/14454393/
14:57 glusterbot Title: #281952 Fedora Project Pastebin (at fpaste.org)
14:57 nbalacha joined #gluster
14:58 shubhendu_ joined #gluster
15:02 * JoeJulian wants to file a bug
15:02 glusterbot https://bugzilla.redhat.com/enter_bug.cgi?product=GlusterFS
15:06 aravindavk joined #gluster
15:06 kdhananjay joined #gluster
15:06 * kkeithley wonders what bug JoeJulian found
15:07 * maserati wants attention
15:09 JoeJulian kkeithley: I've been finding a few.
15:09 JoeJulian This was more of a feature request.
15:11 a_ta_ joined #gluster
15:11 JoeJulian kkeithley: bugs like... libgfapi connects via rrdns hostname. The host it pulled its configuration from is shut down. The VM loses connection and the logs show it continually attempting to connect to the down glusterd instead of using the other dns entry.
15:12 a_ta_ left #gluster
15:12 JoeJulian And that's for a replicated volume.
15:12 JoeJulian That shouldn't happen.
15:13 ccoffey I'm slowly bringing back a downed peer. I updated gluster to 3.6.6. When I try to mount the volume locally, I get fuse: Unknown symbol pagecache_get_page_fixed
15:13 JoeJulian Or the volume that I had replicated, removed one replica that was hosted on a retired server, added a new brick to create a new replica 2 volume. Mount the volume, ls, no files. They're all on the old brick, but the client doesn't see any.
15:14 ivan_rossi left #gluster
15:14 JoeJulian I'm working on the details for each of those and will file the bugs when I have enough information.
15:17 ivan_rossi joined #gluster
15:17 Norky joined #gluster
15:17 ccoffey Would there be an obvious reason for this?
15:17 JoeJulian ccoffey: The only things I can find for that was a (fixed) kernel bug in ubuntu.
15:17 EinstCrazy joined #gluster
15:17 JoeJulian s/ubuntu/debian/
15:17 glusterbot What JoeJulian meant to say was: ccoffey: The only things I can find for that was a (fixed) kernel bug in debian.
15:18 muneerse joined #gluster
15:18 ccoffey hmmm, entirely possible here. That's fine
15:18 JoeJulian Assuming a vanilla kernel, that would mean all of the as yet unidentified kernels of that version.
15:18 JoeJulian Why do people report bugs without referencing versions! Argh!
15:18 ccoffey @joejulian. thanks. I need to reboot after an update, it's an old wheezy kernel currently
15:19 ccoffey I'm just waiting for I assume the intial list of files it needs to heal to finish.
15:19 JoeJulian It'll pick up where it left off.
15:20 ccoffey I have self-heal-daemon on but metadata-self-heal, entry-self-heal and data-self-heal set to off
15:20 JoeJulian So you don't want to mount your client until the heals are done anyway.
15:23 ccoffey it's a 2x4 distribute replicate (i.e. replicate 2, on 8 peers) so I'm still serving data. It'll need to heal 80TB of data, but I don't have the option of downtime
15:23 ccoffey 10 Millions files on the peer set, so ~40 Million in total on the volume
15:24 halloo joined #gluster
15:27 JoeJulian So you have unhealed files served by clients that are set not to check health. So whichever replica they read from, that's the copy that the user gets, modifies, and writes back.
15:28 ccoffey The peer was cleared out and turned back on, so it was empty. The content of the file doesn't change much, so for us that's not an issue. the directory where it is located defines what it is. Some are updated, but it's nothing that'll cause an issue, the conetnts are largely the same
15:29 kshlm joined #gluster
15:29 ccoffey But I do understand your point and for most people this would be an issue
15:29 jbrooks_ joined #gluster
15:30 JoeJulian Cool. I have no problem with people "breaking the rules" as long as they know their own use case. :)
15:31 ccoffey I do have a question though, the load went from 1.0 to about 10 on the "good" replica, glusterfsd. This is with the options I mentioned 11 minutes ago. I'm assuming this is because it's generating the list of files that need healing?
15:33 B21956 joined #gluster
15:33 Lee1092 joined #gluster
15:35 JoeJulian Well, the list is already generated. It's probably creating hash maps to look for the changes within files.
15:35 ctria joined #gluster
15:38 maveric_amitc_ joined #gluster
15:40 ccoffey @JoeJulian: Thanks for the confirmation. I feel a bit more comfortable about this anyway. Prior to the 3.6.6 upgrade (from 3.6.2) this was sending the load to 80
15:40 stickyboy joined #gluster
15:46 rwheeler joined #gluster
15:50 haomaiwang joined #gluster
15:53 EinstCrazy joined #gluster
16:02 maserati Halp? :( http://fpaste.org/281952/14454393/
16:02 glusterbot Title: #281952 Fedora Project Pastebin (at fpaste.org)
16:02 maserati force stop does same thing
16:02 haomaiwa_ joined #gluster
16:05 Pupeno joined #gluster
16:05 JoeJulian at that point I would probably "check the log file for more details". ;)
16:12 maserati Invalid slave name? am i not typing the command right?
16:13 lalatenduM joined #gluster
16:14 hagarth joined #gluster
16:16 maserati This is the log for 3 different ways of trying to type that command: http://fpaste.org/281994/44190144/
16:16 glusterbot Title: #281994 Fedora Project Pastebin (at fpaste.org)
16:18 EinstCrazy joined #gluster
16:22 JoeJulian maserati: volume_name="3par-volume"; slave_url="ssh://utsldl-st-0301::301-volume"; gluster volume geo-replication $volume_name $slave_url stop
16:22 JoeJulian should work
16:23 JoeJulian Usage: volume geo-replication [<VOLNAME>] [<SLAVE-URL>] {create [[no-verify]|[push-pem]] [force]|start [force]|stop [force]|pause [force]|resume [force]|config|status [detail]|delete} [options...]
16:27 maserati http://fpaste.org/282000/44792144/
16:27 glusterbot Title: #282000 Fedora Project Pastebin (at fpaste.org)
16:27 maserati tried both with and without the ssh://
16:31 zhangjn joined #gluster
16:34 brian_ joined #gluster
16:41 JoeJulian stop force maybe?
16:43 rwheeler joined #gluster
16:44 brian_ JoeJulian: How do you determine which of these I want to reference in my repo? http://download.gluster.org/pub/gluster/glusterfs/3.7/LATEST/EPEL.repo/
16:44 glusterbot Title: Index of /pub/gluster/glusterfs/3.7/LATEST/EPEL.repo (at download.gluster.org)
16:45 jbrooks joined #gluster
16:46 maserati JoeJulian: I tried stop force as well
16:46 maserati same result
16:48 jobewan joined #gluster
16:49 ivan_rossi left #gluster
16:56 JoeJulian show me the log entry for that
16:57 shyam left #gluster
17:02 haomaiwa_ joined #gluster
17:02 gmaruzz1 joined #gluster
17:02 gmaruzz1 left #gluster
17:10 kotreshhr joined #gluster
17:12 Rapture joined #gluster
17:12 maserati JoeJulian: http://fpaste.org/282017/45447523/
17:12 glusterbot Title: #282017 Fedora Project Pastebin (at fpaste.org)
17:14 JoeJulian brian_: I don't understand the question.
17:17 JoeJulian maserati: what version is that?
17:18 maserati JoeJulian: glusterfs 3.6.1
17:19 JoeJulian like to live dangerously, don't you. ;)
17:20 JoeJulian Ok, here's the code that's failing: https://github.com/gluster/glusterfs/blob/v3.6.1/xlators/mgmt/glusterd/src/glusterd-geo-rep.c#L1931-L1945
17:20 glusterbot Title: glusterfs/glusterd-geo-rep.c at v3.6.1 · gluster/glusterfs · GitHub (at github.com)
17:21 JoeJulian So glusterd is trying to run gsyncd with some options. Apparently that ocmmand is failing. If you can figure out why, that should solve the problem.
17:21 skylar1 joined #gluster
17:22 maserati oh boy
17:24 jwd_ joined #gluster
17:26 maserati ... any chance upgrading to 3.7 will pull it's head out of it's ass?
17:26 martinet1 joined #gluster
17:26 akik joined #gluster
17:26 a_ta joined #gluster
17:28 kotreshhr joined #gluster
17:29 lalatend1M joined #gluster
17:30 Ramereth joined #gluster
17:34 klaxa joined #gluster
17:47 maserati or maybe will pulling one node out of the gluster peers drop geo-rep ?
17:58 Dan39 joined #gluster
18:02 haomaiwa_ joined #gluster
18:05 cabillman joined #gluster
18:06 jmarley joined #gluster
18:11 a_ta joined #gluster
18:15 18VAAWE9W joined #gluster
18:19 julim_ joined #gluster
18:21 wehde joined #gluster
18:22 ayma joined #gluster
18:30 overclk joined #gluster
18:33 jwd joined #gluster
18:40 brian_ joined #gluster
18:52 kovshenin joined #gluster
18:53 deniszh joined #gluster
19:00 deniszh1 joined #gluster
19:02 haomaiwa_ joined #gluster
19:04 a_ta left #gluster
19:11 kotreshhr left #gluster
19:12 maserati JoeJulian: I can't get that peice of code to straighten up. Is there any other way you can think of ripping geo-rep out? No matter how extreme, i just need to give my super -something- D:
19:43 halloo joined #gluster
20:02 haomaiwa_ joined #gluster
20:03 ayma joined #gluster
20:04 jwaibel joined #gluster
20:16 plarsen joined #gluster
20:27 mpietersen joined #gluster
20:28 mpieters_ joined #gluster
20:29 dlambrig joined #gluster
20:29 mpieters_ joined #gluster
20:30 ctria joined #gluster
20:31 mpieters_ joined #gluster
20:33 linagee joined #gluster
20:33 mpietersen joined #gluster
20:33 linagee can someone please help me take my gluster back to an original install? I'm running debian jessie, I have previous cruft from trying to do gluster, now I'd like to start over.
20:34 wehde has anyone else seen "gluster volume heal vol statistics heal-count" show strange information
20:34 linagee I upgraded from 3.5 to 3.6.6, now glusterfs-server says: "Job for glusterfs-server.service failed. See 'systemctl status glusterfs-server.service' and 'journalctl -xn' for details."
20:34 wehde linagee, did you upgrade all your bricks to 3.6.6?
20:35 linagee wehde: I'd rather just get rid of them all and start over from scratch.
20:35 mpietersen joined #gluster
20:35 linagee can I just remove the glusterfs packages, delete /var/lib/glusterd, then reinstall glusterfs packages?
20:35 linagee (I have all of my data backed up and this is a dev system)
20:35 frozengeek joined #gluster
20:36 wehde linagee, you will have to reformat your bricks
20:36 wehde i would start with gluster volume delete <volume>
20:37 wehde then purge the gluster 3.6.6 packages
20:37 mpietersen joined #gluster
20:37 wehde then install your 3.5 package format your bricks and re add
20:38 linagee Deleting volume will erase all information about the volume. Do you want to continue? (y/n) Connection failed. Please check if gluster daemon is operational.
20:38 linagee ack. :(
20:38 mpietersen joined #gluster
20:38 linagee I think my install is hosed
20:38 wehde how many nodes do you have?
20:38 linagee 2
20:38 wehde did you do the upgrade to both nodes?
20:38 linagee yes. to 3.6.6
20:39 linagee probably absolutely the wrong thing to do, hah.
20:40 wehde did you follow this article?
20:40 wehde http://www.gluster.org/community/documentation/index.php/Upgrade_to_3.6
20:40 linagee I did after the upgrade. :( :(
20:40 linagee I had the heal on and everything.
20:40 wehde if you dont need to save the data because its backed up i would just purge the packages
20:40 wehde install the 3.5 packages
20:41 wehde then format the bricks and re add
20:41 linagee install the 3.5 packages = ?
20:41 wehde you should consider using the latest stable version of gluster
20:41 linagee any reason why not 3.6.6?
20:41 wehde 3.7.xxx
20:41 wehde i would use 3.7 before 3.6
20:41 linagee I'd absolutely definitely use the latest stable version of gluster, but no wheezy support for 3.7
20:42 linagee I will eventually have all of my wheezy boxes upgraded to jessie, but its going to take some weeks.
20:42 wehde i had to upgrade for the "gluster vol heal data split-brain" command
20:42 linagee I'd prefer 3.6.6 over 3.5 as well as I saw the changelog mention some nasty things between them,.
20:42 linagee Not sure if there are many nasty things fixed in 3.6.6 -> 3.7
20:43 wehde i guess if you're not able to use 3.7 probably just go with 3.6.6
20:43 linagee it kind of annoys me there aren't 3.6.6 wheezy packages but I think it said something like "we tried but the dependencies wouldn't work"
20:44 linagee sorry, no wheezy 3.7 packages. :(
20:44 wehde that could be... i'm running 3.7.x on ubuntu 15.04
20:44 linagee A: Wheezy doesn't have new enough versions of some of the dependencies that GlusterFS needs.
20:44 linagee http://download.gluster.org/pub/gluster/glusterfs/3.7/3.7.0/Debian/
20:44 glusterbot Title: Index of /pub/gluster/glusterfs/3.7/3.7.0/Debian (at download.gluster.org)
20:45 rwheeler joined #gluster
20:45 wehde you could add those dependencies manually
20:45 wehde 3.7 fixed a lot of the complaints that i had with gluster
20:46 wehde now if only i could figure out why self heal is never completing
20:46 linagee I'd so love to think that's do-able and that I'd be smarter than the Gluster debian package maintainers. (maybe???)
20:46 wehde does anyone know of a way to force gluster to do a full remirror of a specific brick?
20:48 linagee wehde: looks like libc6 dependency. eww. not about to try that. lol. (I'd bet the gluster package maintainers said the same thing!)
20:49 wehde linagee, i would use 3.5 now that i think about it
20:49 linagee why not 3.6.6?
20:50 wehde linagee, i think there was an issue in 3.6 that was absolutely driving me crazy, cant really remember because it cause me to go to 3.7 right away
20:50 linagee the 3.5->3.6.6 changelogs make me thing 3.5 will literally eat my data and spit it out.
20:52 coreping joined #gluster
20:53 halloo joined #gluster
20:54 dlambrig joined #gluster
20:55 wehde anyone else getting this weird error:
20:55 wehde detected cyclic loop formation during inode linkage. inode
20:57 linagee wehde: what version of gluster?
20:58 wehde linagee, 3.7.3
20:59 dlambrig_ joined #gluster
20:59 linagee wehde: ah good. /me stays away. :-d
20:59 linagee :-D
21:01 jobewan joined #gluster
21:02 haomaiwa_ joined #gluster
21:03 wehde The number of entries in self heal doesn't ever seem to go down
21:17 linagee wehde: hooray! no errors from glusterfs-server anymore. :-D removed gluster packaged, moved /var/lib/glusterd to a temporary location, installed packages, no errors! :-D
21:17 linagee I can haz clean install! done! :-D
21:17 linagee (sorry if that cheezburger cat is annoying to anyone. ;) )
21:20 wehde linagee, awesome now if only i could figure out my problem XD
21:24 brian_ joined #gluster
21:31 atrius` joined #gluster
21:40 stickyboy joined #gluster
21:40 cholcombe joined #gluster
21:47 halloo joined #gluster
22:01 jobewan joined #gluster
22:01 haomaiwa_ joined #gluster
22:07 deniszh joined #gluster
22:07 DV joined #gluster
22:32 ctria joined #gluster
22:33 Pupeno joined #gluster
22:41 dlambrig joined #gluster
22:51 brian_ joined #gluster
23:02 haomaiwa_ joined #gluster
23:23 diegows joined #gluster
23:28 mlhamburg_ joined #gluster
23:31 dlambrig joined #gluster
23:41 gildub joined #gluster
23:52 daMaestro joined #gluster

| Channels | #gluster index | Today | | Search | Google Search | Plain-Text | summary