Welcome to the PiMP OS and Miner.farm forum.

Mixed Rigs, Mining with both AMD and NVIDIA GPUs in the same rig.


  • Staff

    A HOWTO for Mixed Rig Mining


    The PiMP OS has supported mixed GPU rigs for sometime. While the OS has supported any mixture of GPU, each miner is unique. This guide will help you setup your mixed rig for mining a variety of coins We will also cover some tuning tips with included PiMP tools.

    First, one should understand that both NVIDIA and AMD are what we called proprietary, or closed source drivers. If the driver does not support a card type, we have no choice but wait for better support from the vendors. Also be aware that for both AMD and NVIDIA, the Windows and Linux drivers can be out of sync for supported features.

    Not all mixtures of GPUs will work in any given system. The most tested are AMD RX series Polaris 10 & 11, and Nvidia GTX 10x0 series cards (sm60,61). Take some time to research the card selections you are making. Checking with any search engine for Linux support will help a lot in making your GPU choices.

    As well, we currently do not support using Intel video cards. This is possible, but requires a lot of extra effort at all layers. We simply cannot support all the variables here. If you wish to try it yourself, just keep in mind that the Intel cards will also need to be excluded by the miners. It really is best to simply disable onboard or Intel video cards.

    Nvidia still uses the X11 drivers for support of monitoring and maintenance. This means you will need to use the Nvidia cards first on your build. Read the manuals for your MB for the best reference on slot placement. For the most part, you can simply use the first x16 slot on the board. You will find the best support with all drivers and miners by grouping all your Nvidia cards in order. So they are the 1st, 2nd, 3rd.. etc.. cards. Put the AMD cards into a group after.

    You can use lspci -vt to get a nice tree like view of how the OS sees your build. Make sure they cascade out in groups of Nvidia first, and AMD next.

    [/root]:# lspci -vt
    -[0000:00]-+-00.0  Advanced Micro Devices, Inc. [AMD/ATI] RD9x0/RX980 Host Bridge
               +-00.2  Advanced Micro Devices, Inc. [AMD/ATI] RD890S/RD990 I/O Memory Management Unit (IOMMU)
               +-02.0-[01-04]----00.0-[02-04]--+-08.0-[03]--+-00.0  NVIDIA Corporation GK104 [GeForce GTX 680]
               |                               |            \-00.1  NVIDIA Corporation GK104 HDMI Audio Controller
               |                               \-10.0-[04]--+-00.0  NVIDIA Corporation GP106 [GeForce GTX 1060 3GB]
               |                                            \-00.1  NVIDIA Corporation GP106 High Definition Audio Controller
               +-04.0-[05]----00.0  ASMedia Technology Inc. ASM1062 Serial ATA Controller
               +-05.0-[06]----00.0  ASMedia Technology Inc. ASM1062 Serial ATA Controller
               +-09.0-[07]----00.0  ASMedia Technology Inc. ASM1042 SuperSpeed USB Host Controller
               +-0a.0-[08]--+-00.0  Advanced Micro Devices, Inc. [AMD/ATI] Baffin [Radeon RX 460/560D / Pro 450/455/460/560]
               |            \-00.1  Advanced Micro Devices, Inc. [AMD/ATI] Device aae0
               +-0b.0-[09]--+-00.0  Advanced Micro Devices, Inc. [AMD/ATI] Baffin [Radeon RX 460/560D / Pro 450/455/460/560]
               |            \-00.1  Advanced Micro Devices, Inc. [AMD/ATI] Device aae0
               +-11.0  Advanced Micro Devices, Inc. [AMD/ATI] SB7x0/SB8x0/SB9x0 SATA Controller [AHCI mode]
               +-12.0  Advanced Micro Devices, Inc. [AMD/ATI] SB7x0/SB8x0/SB9x0 USB OHCI0 Controller
               +-12.2  Advanced Micro Devices, Inc. [AMD/ATI] SB7x0/SB8x0/SB9x0 USB EHCI Controller
               +-13.0  Advanced Micro Devices, Inc. [AMD/ATI] SB7x0/SB8x0/SB9x0 USB OHCI0 Controller
               +-13.2  Advanced Micro Devices, Inc. [AMD/ATI] SB7x0/SB8x0/SB9x0 USB EHCI Controller
               +-14.0  Advanced Micro Devices, Inc. [AMD/ATI] SBx00 SMBus Controller
               +-14.2  Advanced Micro Devices, Inc. [AMD/ATI] SBx00 Azalia (Intel HDA)
               +-14.3  Advanced Micro Devices, Inc. [AMD/ATI] SB7x0/SB8x0/SB9x0 LPC host controller
               +-14.4-[0a]--
               +-14.5  Advanced Micro Devices, Inc. [AMD/ATI] SB7x0/SB8x0/SB9x0 USB OHCI2 Controller
               +-15.0-[0b]--
               +-15.1-[0c]----00.0  Realtek Semiconductor Co., Ltd. RTL8111/8168/8411 PCI Express Gigabit Ethernet Controller
               +-15.2-[0d]----00.0  ASMedia Technology Inc. ASM1042 SuperSpeed USB Host Controller
               +-15.3-[0e]----00.0  ASMedia Technology Inc. ASM1042 SuperSpeed USB Host Controller
               +-16.0  Advanced Micro Devices, Inc. [AMD/ATI] SB7x0/SB8x0/SB9x0 USB OHCI0 Controller
               +-16.2  Advanced Micro Devices, Inc. [AMD/ATI] SB7x0/SB8x0/SB9x0 USB EHCI Controller
               +-18.0  Advanced Micro Devices, Inc. [AMD] Family 15h Processor Function 0
               +-18.1  Advanced Micro Devices, Inc. [AMD] Family 15h Processor Function 1
               +-18.2  Advanced Micro Devices, Inc. [AMD] Family 15h Processor Function 2
               +-18.3  Advanced Micro Devices, Inc. [AMD] Family 15h Processor Function 3
               +-18.4  Advanced Micro Devices, Inc. [AMD] Family 15h Processor Function 4
               \-18.5  Advanced Micro Devices, Inc. [AMD] Family 15h Processor Function 5
    

    In this example, you can see two differing Nvidia cards, and two AMD cards setup so the Nvidia is first, and AMD second. This configuration will work with the most miners with the least amount of work.

    We can see the results of this with gputool also:

    [/root]:# gputool --list
    PIMP GPU Tool v20180220 
    GPUs type: Mixed 
     ---
    4 GPUs seen in hardware:
    03:00.0 VGA compatible controller: NVIDIA Corporation GK104 [GeForce GTX 680] (rev a1) (prog-if 00 [VGA controller])
    04:00.0 VGA compatible controller: NVIDIA Corporation GP106 [GeForce GTX 1060 3GB] (rev a1) (prog-if 00 [VGA controller])
    08:00.0 VGA compatible controller: Advanced Micro Devices, Inc. [AMD/ATI] Baffin [Radeon RX 460/560D / Pro 450/455/460/560] (rev e5) (prog-if 00 [VGA controller])
    09:00.0 VGA compatible controller: Advanced Micro Devices, Inc. [AMD/ATI] Baffin [Radeon RX 460/560D / Pro 450/455/460/560] (rev e5) (prog-if 00 [VGA controller])
     ---
    GPUs reported by AMD driver:
    GPU:08.00.0:Radeon RX 460:115-C994PI0-102:Micron MT51J256M3:GDDR5:Polaris11
    GPU:09.00.0:Radeon RX 460:115-C994PI0-102:Micron MT51J256M3:GDDR5:Polaris11
     ---
    GPUs reported by NVidia driver:
    GPU 0: GeForce GTX 680 (UUID: GPU-d9e11478-834d-ea1f-d7df-0316b656e708)
    GPU 1: GeForce GTX 1060 3GB (UUID: GPU-7585f78a-8d28-faa4-af25-851b471a56bc)
    

    At this point the Claymore-dual miner should work just fine. Make sure you pimpup first! You can install it for either AMD or NVIDIA:

    [/root]:# pimp --add 1651
    Miner ID 9 added
    bin: /opt/miners/claymore-dual/ethdcrminer64 
    config: /opt/confs/claymore.dual.pcfg 
    address: 127.0.0.1:4036
    This miner can be started with 'pimp --start 9'
    

    First time, use debug mode. Please ignore the fact that some of these cards do not have enough memory for the DAG :) Just know that you should always be aware of the mining requirements of the coin you want to mine!

    [/root]:# pimp --debug 9
    Starting miner ID 9 without screen for debugging. 
    For normal operation, stop the miner using Control+C, 
    then start it using 'pimp --start 9' or the minerfarm agent console.
    Executing: /opt/miners/claymore-dual/ethdcrminer64 -epool stratum+tcp://us1.ethermine.org:4444 -ewal 0xe8C128B81B26bA1614f4CF2292A5D754Ca352c91.PimpWorker -epsw x  -allpools 1 -eworker Pimpworker -tt 75 -tstop 90 -dbg 1
    
    ????????????????????????????????????????????????????????????????ͻ
    ?                Claymore's Dual GPU Miner - v11.1               ?
    ?              ETH + DCR/SIA/LBC/PASC/BLAKE2S/KECCAK             ?
    ????????????????????????????????????????????????????????????????ͼ
    
    ETH: 1 pool is specified
    Main Ethereum pool is us1.ethermine.org:4444
    DCR: 0 pool is specified
    At least 16 GB of Virtual Memory is required for multi-GPU systems
    Make sure you defined GPU_MAX_ALLOC_PERCENT 100
    Be careful with overclocking, use default clocks for first tests
    Press "s" for current statistics, "0".."9" to turn on/off cards, "r" to reload pools, "e" or "d" to select current pool 
    OpenCL initializing...
    
    AMD Cards available: 2 
    GPU #0: Baffin (Radeon RX 560 Series), 2184 MB available, 14 compute units (pci bus 8:0:0)
    GPU #0 recognized as Radeon RX 460/560
    GPU #1: Baffin (Radeon RX 560 Series), 2184 MB available, 14 compute units (pci bus 9:0:0)
    GPU #1 recognized as Radeon RX 460/560
    POOL/SOLO version
    GPU #0: algorithm ASM
    GPU #1: algorithm ASM
    Be careful with overclocking, use default clocks for first tests
    Press "s" for current statistics, "0".."9" to turn on/off cards, "r" to reload pools, "e" or "d" to select current pool 
    CUDA initializing...
    
    NVIDIA Cards available: 2 
    CUDA Driver Version/Runtime Version: 9.1/8.0
    GPU #2: GeForce GTX 680, 1996 MB available, 8 compute units, capability: 3.0  (pci bus 3:0:0)
    
    GPU #3: GeForce GTX 1060 3GB, 3019 MB available, 9 compute units, capability: 6.1  (pci bus 4:0:0)
    
    Total cards: 4 
    
    Some GPUs (3) are 2GB and some (1) are not, developer fee will not be mined on 2GB GPUs, so devfee will be mined on 1 cards of 4.
    See Readme for details.
    
    AMD ADL library not found.
    ETH: Stratum - connecting to 'us1.ethermine.org' <18.219.59.155> port 4444 (unsecure)
    ETH: Stratum - Connected (us1.ethermine.org:4444) (unsecure)
    No pool specified for Decred! Ethereum-only mining mode is enabled
    ETHEREUM-ONLY MINING MODE ENABLED (-mode 1)
    ETH: eth-proxy stratum mode
    "-allpools" option is set, default pools can be used for devfee, check "Readme" file for details.
    Watchdog enabled
    Remote management (READ-ONLY MODE) is enabled on port 3333
    
    ETH: Authorized
    Setting DAG epoch #178...
    Setting DAG epoch #178 for GPU3
    Setting DAG epoch #178 for GPU0
    Setting DAG epoch #178 for GPU2
    Create GPU buffer for GPU0
    Create GPU buffer for GPU3
    Setting DAG epoch #178 for GPU1
    

    Very few miner applications support mixed rig mining. They are separated by GPU maker in PiMP for a reason. For those cases, you simply run one miner for each GPU type. Keep in mind that not all mining application support the commands required to select which GPUs it might use. For an example, let's look at ethminer.

    First thing, is we install the miner using pimp --add as normal for any miner. Next we look at the options for the miner, for each miner this is different. Best to refer to the miners web help guides. For ethminer, just just run it like this:

    [/opt/miners/ethminer]:# ./ethminer --help
    Usage ethminer [OPTIONS]
    Options:
    
    Work farming mode:
        -F,--farm <url>  (deprecated) Put into mining farm mode with the work server at URL (default: http://127.0.0.1:8545)
        -FF,-FO, --farm-failover, --stratum-failover <url> (deprecated) Failover getwork/stratum URL (default: disabled)
    	--farm-retries <n> Number of retries until switch to failover (default: 3)
    	-S, --stratum <host:port>  (deprecated) Put into stratum mode with the stratum server at host:port
    	-SF, --stratum-failover <host:port>  (deprecated) Failover stratum server at host:port
        -O, --userpass <username.workername:password> (deprecated) Stratum login credentials
        -FO, --failover-userpass <username.workername:password> (deprecated) Failover stratum login credentials (optional, will use normal credentials when omitted)
        --work-timeout <n> reconnect/failover after n seconds of working on the same (stratum) job. Defaults to 180. Don't set lower than max. avg. block time
        --stratum-ssl [<n>]  (deprecated) Use encryption to connect to stratum server.
            0: Force TLS1.2 (default)
            1: Allow any TLS version
            2: Allow self-signed or invalid certs and any TLS version
        -SP, --stratum-protocol <n> (deprecated) Choose which stratum protocol to use:
            0: official stratum spec: ethpool, ethermine, coinotron, mph, nanopool (default)
            1: eth-proxy compatible: dwarfpool, f2pool, nanopool (required for hashrate reporting to work with nanopool)
            2: EthereumStratum/1.0.0: nicehash
        -RH, --report-hashrate Report current hashrate to pool (please only enable on pools supporting this)
        -HWMON [<n>], Displays gpu temp, fan percent and power usage. Note: In linux, the program uses sysfs, which may require running with root priviledges.
            0: Displays only temp and fan percent (default)
            1: Also displays power usage
        --exit Stops the miner whenever an error is encountered
        -SE, --stratum-email <s> Email address used in eth-proxy (optional)
        --farm-recheck <n>  Leave n ms between checks for changed work (default: 500). When using stratum, use a high value (i.e. 2000) to get more stable hashrate output
        -P URL Specify a pool URL. Can be used multiple times. The 1st for for the primary pool, and the 2nd for the failover pool.
            URL takes the form: scheme://user[:password]@hostname:port.
            for getwork use one of the following schemes:
              http
            for stratum use one of the following schemes: 
              stratum+ssl stratum+tcp stratum+tls stratum+tls12 stratum1+ssl stratum1+tcp stratum1+tls stratum1+tls12 stratum2+ssl stratum2+tcp stratum2+tls stratum2+tls12
            Example: stratum+ssl://0x012345678901234567890234567890123.miner1@ethermine.org:5555
    
    Benchmarking mode:
        -M [<n>],--benchmark [<n>] Benchmark for mining and exit; Optionally specify block number to benchmark against specific DAG.
        --benchmark-warmup <seconds>  Set the duration of warmup for the benchmark tests (default: 3).
        --benchmark-trial <seconds>  Set the duration for each trial for the benchmark tests (default: 3).
        --benchmark-trials <n>  Set the number of benchmark trials to run (default: 5).
    Simulation mode:
        -Z [<n>],--simulation [<n>] Mining test mode. Used to validate kernel optimizations. Optionally specify block number.
    Mining configuration:
        -G,--opencl  When mining use the GPU via OpenCL.
        -U,--cuda  When mining use the GPU via CUDA.
        -X,--cuda-opencl Use OpenCL + CUDA in a system with mixed AMD/Nvidia cards. May require setting --opencl-platform 1 or 2. Use --list-devices option to check which platform is your AMD. 
        --opencl-platform <n>  When mining using -G/--opencl use OpenCL platform n (default: 0).
        --opencl-device <n>  When mining using -G/--opencl use OpenCL device n (default: 0).
        --opencl-devices <0 1 ..n> Select which OpenCL devices to mine on. Default is to use all
        -t, --mining-threads <n> Limit number of CPU/GPU miners to n (default: use everything available on selected platform)
        --list-devices List the detected OpenCL/CUDA devices and exit. Should be combined with -G, -U, or -X flag
        --display-interval <n> Set mining stats display interval in seconds. (default: every 5 seconds)
        -L, --dag-load-mode <mode> DAG generation mode.
            parallel    - load DAG on all GPUs at the same time (default)
            sequential  - load DAG on GPUs one after another. Use this when the miner crashes during DAG generation
            single <n>  - generate DAG on device n, then copy to other devices
     OpenCL configuration:
        --cl-kernel <n>  Use a different OpenCL kernel (default: use stable kernel)
            0: stable kernel
            1: experimental kernel
        --cl-local-work Set the OpenCL local work size. Default is 128
        --cl-global-work Set the OpenCL global work size as a multiple of the local work size. Default is 8192 * 128
        --cl-parallel-hash <1 2 ..8> Define how many threads to associate per hash. Default=8
     CUDA configuration:
        --cuda-block-size Set the CUDA block work size. Default is 128
        --cuda-grid-size Set the CUDA grid size. Default is 8192
        --cuda-streams Set the number of CUDA streams. Default is 2
        --cuda-schedule <mode> Set the schedule mode for CUDA threads waiting for CUDA devices to finish work. Default is 'sync'. Possible values are:
            auto  - Uses a heuristic based on the number of active CUDA contexts in the process C and the number of logical processors in the system P. If C > P, then yield else spin.
            spin  - Instruct CUDA to actively spin when waiting for results from the device.
            yield - Instruct CUDA to yield its thread when waiting for results from the device.
            sync  - Instruct CUDA to block the CPU thread on a synchronization primitive when waiting for the results from the device.
        --cuda-devices <0 1 ..n> Select which CUDA GPUs to mine on. Default is to use all
        --cuda-parallel-hash <1 2 ..8> Define how many hashes to calculate in a kernel, can be scaled to achieve better performance. Default=4
        --cuda-noeval  bypass host software re-evalution of GPU solutions.
            This will trim some milliseconds off the time it takes to send a result to the pool.
            Use at your own risk! If GPU generates errored results they WILL be forwarded to the pool
            Not recommended at high overclock.
     API core configuration:
        --api-port Set the api port, the miner should listen to. Use 0 to disable. Default=0, use negative numbers to run in readonly mode. for example -3333.
     General Options:
        -v,--verbosity <0 - 9>  Set the log verbosity from 0 to 9 (default: 5). Set to 9 for switch time logging.
        -V,--version  Show the version and exit.
        -h,--help  Show this help message and exit.
     Envionment variables:
         NO_COLOR - set to any value to disable color output. Unset to re-enable color output.
    

    Holy Smokes! Thats a lot of options. We can see though that it supports both AMD and NVIDIA. As of the writing of this article, it supports ether AMD or NVIDIA. Both can be a problem with some rigs, pools, etc.

    We look for an option in the miner that lists the devices or GPUs. This will give us the best idea of what it can be made to support. For this miner, we type ./ethminer --list-devices

    [/opt/miners/ethminer]:# ./ethminer --list-devices
    
    Listing OpenCL devices.
    FORMAT: [platformID] [deviceID] deviceName
    [0] [0] GeForce GTX 1060 3GB
    	CL_DEVICE_TYPE: GPU
    	CL_DEVICE_GLOBAL_MEM_SIZE: 3166109696
    	CL_DEVICE_MAX_MEM_ALLOC_SIZE: 791527424
    	CL_DEVICE_MAX_WORK_GROUP_SIZE: 1024
    [0] [1] GeForce GTX 680
    	CL_DEVICE_TYPE: GPU
    	CL_DEVICE_GLOBAL_MEM_SIZE: 2093809664
    	CL_DEVICE_MAX_MEM_ALLOC_SIZE: 523452416
    	CL_DEVICE_MAX_WORK_GROUP_SIZE: 1024
    [1] [0] Baffin
    	CL_DEVICE_TYPE: GPU
    	CL_DEVICE_GLOBAL_MEM_SIZE: 2290253824
    	CL_DEVICE_MAX_MEM_ALLOC_SIZE: 2025947136
    	CL_DEVICE_MAX_WORK_GROUP_SIZE: 256
    [1] [1] Baffin
    	CL_DEVICE_TYPE: GPU
    	CL_DEVICE_GLOBAL_MEM_SIZE: 2290253824
    	CL_DEVICE_MAX_MEM_ALLOC_SIZE: 2025947136
    	CL_DEVICE_MAX_WORK_GROUP_SIZE: 256
    
    Listing CUDA devices.
    FORMAT: [deviceID] deviceName
    [0] GeForce GTX 1060 3GB
    	Compute version: 6.1
    	cudaDeviceProp::totalGlobalMem: 3166109696
    	Pci: 0000:04:00
    [1] GeForce GTX 680
    	Compute version: 3.0
    	cudaDeviceProp::totalGlobalMem: 2093809664
    	Pci: 0000:03:00
    

    So, lets look at the profile for ethminer now. Focus on the FLAGS section:

    # ---------------------------------------------------------------------
    # COMMAND LINE PARAMETERS:
    # Here is where you can add command line parameters such as intensity and mining modes.
    # NOTE: the pool/wallet/algo settings are set by PIMP. Set those above.
    
    FLAGS=-G --opencl-platform 1 --farm-recheck 2000
    

    From the help screen for ethminer, we can see this means 'Mine with AMD, using the opencl platform type 1'. Combine that with the output of list-devices, and we can see that hardware platform 1 is the two Baffin cards in this rig. If you had only AMD cards, the opencl-platform would be 0. If your Intel card was still enabled (remember, not supported), the opencl-platform would be 3 :)

    Since NVIDIA uses CUDA, you would change the -G to a -U to mine on the Nvidia cards. Keep in mind you will need to be using PiMP 2.5.1 at a minimum for ethminer with Nvidia support.

    You can also try replacing -G or -U with -X, just remember that at this time we are not supporting ethminer with the -X option. Course, just because we cannot officially support it, does not mean you cannot try it to see what happens. The worst that can happen using -X is your rig can hang. Please remember that the alias rb should be used to reboot. With ethminer, I find it can lock up so CTRL-C no longer works. Instead, bail out of the screen session or use the suspend key CTRL-Z by default. You can then kill the job, or rb the rig.

    Once you setup one miner instance for AMD, you can install another for NVIDIA. Just pimp add another instance. Edit it up just like the AMD one, make sure one calls -G and sets the opencl-platform correctly. And the other calls -U for NVIDIA. Then start both miners using pimp --start as normal.

    You could do this using any two miners. Simply look up the miners method for selecting one type of GPU over another, and add it to the FLAGS variable in a profile. Use the miners device listing function to understand how they are mapped. Sometimes a miner might call that --ndevs or -N, just read the miner help or docs closely.

    Currently, there are some licensing considerations with this method and Miner.farm. Be aware that every running miner profile, uses a Miner.farm license at this time. Also understand that GPU indexing in Miner.farm is sometimes dependant on the miner statistics. So statistics on Miner.farm may be skewed. We are working to modify that for future, but mixed gpu rigs are a new idea so please be patient and understand the limitations.

    I will update with other combinations of miners in the future.

    Cheers
    Anjin


  • Staff

    Brilliant work Anjin, and an amazing write-up!!!


 



Want 10% more hash from your rigs?



We promise to keep your email safe and never spam you.



Copyright (c) 2012-2018 PiMP LLC. All rights Reserved.