| Server IP : 35.80.110.71 / Your IP : 216.73.216.221 Web Server : Apache/2.4.58 (Ubuntu) System : Linux ip-172-31-21-44 6.17.0-1019-aws #19~24.04.1-Ubuntu SMP Tue Jun 23 18:53:06 UTC 2026 x86_64 User : ubuntu ( 1000) PHP Version : 8.3.31 Disable Function : NONE MySQL : OFF | cURL : ON | WGET : ON | Perl : ON | Python : OFF | Sudo : ON | Pkexec : OFF Directory : /lib/python3/dist-packages/scipy/stats/__pycache__/ |
Upload File : |
�
x�f�8 � �\ � d dl mZ d dlZd dlZd dlmZmZmZ d dl m
Z
d dl Z d dl m
Z
mZ d dlmZmZmZmZ d dl mZ d dlmZ d d
lmZ d d lmZmZ d d l m Z g d
�Z d� Z d9d�Z d:d�Z!d� Z"d� Z#d� Z$d� Z%e G d� d� � Z&ddddd dddddd �
d �Z'd � Z(e G d � d � � Z) ed!d"� ddddd d#�d$�� Z*e G d%� d&� � Z+d'� Z,d(� Z-d)� Z. d9d*�Z/ d9d+�Z0 d9d,�Z1d-� Z2d.ddddd dd/�d0�Z3e G d1� d2� � Z4e G d3� d4e4� � Z5e G d5� d6e4� � Z6e G d7� d8e4� � Z7y);� )� annotationsN)� combinations� permutations�product)�Sequence)�check_random_state�_rename_parameter)�ndtr�ndtri�comb� factorial)� rng_integers)� dataclass� )�ConfidenceInterval)�_broadcast_concatenate�_broadcast_arrays)�DegenerateDataWarning)� bootstrap�monte_carlo_test�permutation_testc � � � dd�� fd�
}|S )z Vectorize an n-sample statisticr ��axisc � ��� |D �cg c] }|j | �� }}t j |� d d �t || � }t j || d� }��fd�}t j
|d|� d S c c}w )N���r c �: �� t j | �� } �|� S �N)�np�split)�z�data�
split_indices� statistics ���9/usr/lib/python3/dist-packages/scipy/stats/_resampling.py�stat_1dz6_vectorize_statistic.<locals>.stat_nd.<locals>.stat_1d" s �� ��8�8�A�}�-�D� �d�#� #� � )�shaper �cumsumr �moveaxis�apply_along_axis)r r" �sample�lengthsr! r& r# r$ s @�r% �stat_ndz%_vectorize_statistic.<locals>.stat_nd sz �� �48�9�&�6�<�<��%�9��9�� � �'�*�3�B�/�
� "�4�� .��
�K�K��4�� #�� $� �"�"�7�A�q�1�"�5�5�� :s �A:r( )r$ r/ s ` r% �_vectorize_statisticr0 s �� � � 6� �Nr' c # � K � | j d }|xs |}t d||� D ]� }t |||z
� }t j ||ft
�� }t j |dd�|||z �f d� t j |� }t j |||f� }|| j ||dz
f� }| d|f }|�� �� y�w)z=Jackknife resample the sample. Only one-sample stats for now.r r ��dtypeNFr .)
r) �range�minr �ones�bool�
fill_diagonal�arange� broadcast_to�reshape) r- �batch�n�
batch_nominal�k� batch_actual�j�i� resampless r% �_jackknife_resamplerD * s� � �� �� � �R��A��J�Q�M�
�1�a� �
'� ���=�!�A�#�.� �
�G�G�\�1�%�T� 2��
����1�a��,�� .�.�/��7� �I�I�a�L�� �O�O�A� �a� 0� 1��
�a�D�L�L�,��!��,� -���3��6�N� ��� �s �CC
c �R � | j d }t |d|||f� }| d|f }|S )z Bootstrap resample the sample.r r .)r) r )r- � n_resamples� random_stater= rB rC s r% �_bootstrap_resamplerH > s: � �� � �R��A� �\�1�a�+�q�)9�:�A��s�A�v��I� �r' c � � | j | }| |k j |�� | |k j |�� z d|z z S )z�Vectorized, simplified `scipy.stats.percentileofscore`.
Uses logic of the 'mean' value of percentileofscore's kind parameter.
Unlike `stats.percentileofscore`, the percentile returned is a fraction
in [0, 1].
r � )r) �sum)�a�scorer �Bs r% �_percentile_of_scorerO I sF � �
����
�A�
��Y�O�O��O� &�!�u�*�)9�)9�t�)9�)D� D��Q�� O�Or' c � � | j dd }t j ||� }t j |t j �� }t j
|� D ]l \ }}t j |� r4d}t j t |� � t j ||< �O| | }t j ||� ||< �n |d S )z9`np.percentile` with different percentile for each slice.Nr r2 z�The BCa confidence interval cannot be calculated. This problem is known to occur when the distribution is degenerate or the statistic is np.min.r( ) r) r r: �
zeros_like�float64� ndenumerate�isnan�warnings�warnr �nan�
percentile)� theta_hat_b�alphar) � percentiles�indices�alpha_i�msg�
theta_hat_b_is r% �_percentile_along_axisr` T s� � �
� � �c�r� "�E� �O�O�E�5� )�E��-�-��R�Z�Z�8�K� �N�N�5� 1� I����
�8�8�G� �=�
�
�M�M�/��4� 5�#%�6�6�K�� � '�� 0�M�#%�=�=� ��#H�K�� � I� �r�?�r' c � � t j || d|i�� d }t ||d�� }t |� }g } t | � D ]u \ }
} | D � cg c] } t j
| d� �� } } g }
t
|� D ]+ }|| |
< t | d�� }|
j ||ddi�� �- | j |
� �w | D �
cg c] }
t j |
d�� �� } }
| D �
cg c] }
|
j d �� }}
| D �
cg c] }
|
j dd�� �� }}
t || |� D ��
�cg c] \ }}
}|dz
||
z
z �� }}
}}t ||� D ��cg c] \ }}|d z j d�� |d z z ��" }}}t ||� D ��cg c] \ }}|d
z j d�� |d
z z ��" }}}d t |� z t |� d z z }t |� }| }||z }t ||d||z z
z z � } ||z } t || d|| z z
z z � } | | |fS c c} w c c}
w c c}
w c c}
w c c}}
}w c c}}w c c}}w )
z(Bias-corrected and accelerated interval.r ).Nr r ���T)r �keepdimsr � rJ gUUUUUU�?g �?)r �asarrayrO r � enumerate� expand_dimsrD r �append� concatenater) �mean�ziprK r
) r" r$ r rZ rY r< � theta_hatrX �z0_hat� theta_hat_jirA r- �samples� theta_hat_i�jackknife_sample� broadcasted�n_j�theta_hat_j_dot�
theta_hat_dotr= �U_ji�U_i�nums�dens�a_hat�z_alpha�z_1alpha�num1�alpha_1�num2�alpha_2s r% �
_bca_intervalr� l s� � �
�
�
�9�d�6��6�7� �B�I�%�k�9�2�F�J�
�:�
�F� �L� �t�_� )� ��6� =A�A�&�2�>�>�&�"�-�A��A�� � 3�F�E� B� A� �)�G�A�J�+�G�"�=�K� � � �y�+� ?�B� ?� @� A� ���K�(� )� (4�5�#� �N�N�;�R�8� 5�L� 5� 5A�
A�[�;� � �R� �
A�C�
A� +7�8�&� #�'�'�R�$�'�?� 8�O� 8�
�?�L�#�6� 8� 8�-�
�{�A� ��U�}�{�2� 3� 8�D� 8� 69��s�^� D�6�3��S�!�V�L�L�b�L� !�!�Q�$� &� D�D� D�58��s�^� D�6�3��S�!�V�L�L�b�L� !�!�Q�$� &� D�D� D� �#�d�)�O�c�$�i�#� .� .�E� �E�l�G��x�H� �G� �D��6�D�!�e�D�j�.� 1�1�2�G� �H� �D��6�D�!�e�D�j�.� 1�1�2�G� �G�U� "�"��? B��5�� B��8�� 8�� E�� Ds* � H � H#�%H(�H-�.H2�%H9�%H?c � � |dvr t d� �|�!dt j |� j v }|s t |� }t |� } || k7 r t d� �d}
t
| � }
|
dk( r t d� �g }| D ]\ }t j |� }|j | d k r t d
� �t j || d � }|j |� �^ |d vr t d
� �|rX|d j d }|d d D ]! }|j d |k7 s�d}t |� � d ||fd�}t j |� g}t |� }|j � }h d�}||vrt d|� �� �t |� }||k7 s|dk r t d� �|�|}n t |� }||k7 s|dk r t d� �h d�}| j � } | |vrt d|� �� �d}|
�t! |
d� s t |� �d}|
r|
j" j$ s|dk( r t |� �t' | � } ||||| ||||| |
| f S # t $ r t d� �w xY w)z5Input validation and standardization for `bootstrap`.� FNT�0`vectorized` must be `True`, `False`, or `None`.Nr �`axis` must be an integer.r z%`data` must be a sequence of samples.z(`data` must contain at least one sample.r �Ieach sample in `data` must contain two or more observations along `axis`.r > FTz#`paired` must be `True` or `False`.zIWhen `paired is True`, all samples must have the same length along `axis`c �D � |D �cg c] }|d| f �� }} ||d|i�S c c}w )N.r r( )rB r r" �unpaired_statisticr- s r% r$ z _bootstrap_iv.<locals>.statistic� s2 � �15�6�v�F�3��6�N�6�D�6�%�t�7�$�7� 7�� 7s � > �less�greater� two-sidedz `alternative` must be one of z-`n_resamples` must be a non-negative integer.�+`batch` must be a positive integer or None.> �bca�basicrX z`method` must be in z?`bootstrap_result` must have attribute `bootstrap_distribution'�bootstrap_distributionzXEither `bootstrap_result.bootstrap_distribution.size` or `n_resamples` must be positive.)�
ValueError�inspect� signature�
parametersr0 �int�len� TypeErrorr �
atleast_1dr) r+ rh r9 �float�lower�hasattrr� �sizer )r" r$ �
vectorized�pairedr �confidence_level� alternativerF r< �method�bootstrap_resultrG �axis_int� n_samples�data_ivr- r= �message�confidence_level_float� alternatives�n_resamples_int�batch_iv�methodss r% �
_bootstrap_ivr� � s� � �
�,�,��K�L�L����w� 0� 0�� ;� F� F�F�
� �(��3� ��4�y�H� �x���5�6�6��I�B���I� � �A�~��C�D�D��G�� ��� � �v�&�� �<�<�� !�Q� &� � :� ;�
;����V�X�r�2�����v� �
� �]�"��>�?�?�
� �A�J� � �R� �� �a�b�k� *�F��|�|�B� �1�$� 6�� ��)�)� *� !�w�9� 8� �9�9�Q�<�.�� "�#3� 4�� �#�#�%�K�3�L��,�&��8� ��G�H�H��+�&�O��o�%� �1�)<��H�I�I� �}����u�:�� �H� ��A�
� � J�K� K�,�G�
�\�\�^�F�
�W� ��/��y�9�:�:�O�G��$�� ,�.F�G���!�!�1�G�
�
�
4�
4�
9�
9� �1�$���!�!�%�l�3�L� �Y�
�F�H� "�K� �(� �$�l� 4� 4��O � B��@�A�A�B�s � H? �?Ic �0 � e Zd ZU dZded< ded< ded< y) �BootstrapResulta� Result object returned by `scipy.stats.bootstrap`.
Attributes
----------
confidence_interval : ConfidenceInterval
The bootstrap confidence interval as an instance of
`collections.namedtuple` with attributes `low` and `high`.
bootstrap_distribution : ndarray
The bootstrap distribution, that is, the value of `statistic` for
each resample. The last dimension corresponds with the resamples
(e.g. ``res.bootstrap_distribution.shape[-1] == n_resamples``).
standard_error : float or ndarray
The bootstrap standard error, that is, the sample standard
deviation of the bootstrap distribution.
r �confidence_interval�
np.ndarrayr� �float | np.ndarray�standard_errorN��__name__�
__module__� __qualname__�__doc__�__annotations__r( r' r% r� r� s � �� ,�+� &�&�&�&r' r� �' Fgffffff�?r� �BCa)
rF r< r� r� r r� r� r� r� rG c
�b � t | ||||||||| |
| � } | \ } }}}}}}}}} }
} |
�g n |
j g}
|xs |xs d}t d||� D ]P }t |||z
� }g }| D ]! }t ||| �� }|j |� �# |
j ||ddi�� �R t
j |
d�� }
|dk( rd|z
d z nd|z
}| d
k( rt | |d||
|� � dd }t }n
|d|z
f}d � } ||
|d d
z � } ||
|d d
z � }| dk( r || ddi�}d |z |z
d |z |z
}}|dk( r&t
j |t j � }n)|dk( r$t
j |t j � }t t ||� |
t
j |
dd�� �� S )a�8
Compute a two-sided bootstrap confidence interval of a statistic.
When `method` is ``'percentile'`` and `alternative` is ``'two-sided'``,
a bootstrap confidence interval is computed according to the following
procedure.
1. Resample the data: for each sample in `data` and for each of
`n_resamples`, take a random sample of the original sample
(with replacement) of the same size as the original sample.
2. Compute the bootstrap distribution of the statistic: for each set of
resamples, compute the test statistic.
3. Determine the confidence interval: find the interval of the bootstrap
distribution that is
- symmetric about the median and
- contains `confidence_level` of the resampled statistic values.
While the ``'percentile'`` method is the most intuitive, it is rarely
used in practice. Two more common methods are available, ``'basic'``
('reverse percentile') and ``'BCa'`` ('bias-corrected and accelerated');
they differ in how step 3 is performed.
If the samples in `data` are taken at random from their respective
distributions :math:`n` times, the confidence interval returned by
`bootstrap` will contain the true value of the statistic for those
distributions approximately `confidence_level`:math:`\, \times \, n` times.
Parameters
----------
data : sequence of array-like
Each element of data is a sample from an underlying distribution.
statistic : callable
Statistic for which the confidence interval is to be calculated.
`statistic` must be a callable that accepts ``len(data)`` samples
as separate arguments and returns the resulting statistic.
If `vectorized` is set ``True``,
`statistic` must also accept a keyword argument `axis` and be
vectorized to compute the statistic along the provided `axis`.
n_resamples : int, default: ``9999``
The number of resamples performed to form the bootstrap distribution
of the statistic.
batch : int, optional
The number of resamples to process in each vectorized call to
`statistic`. Memory usage is O(`batch`*``n``), where ``n`` is the
sample size. Default is ``None``, in which case ``batch = n_resamples``
(or ``batch = max(n_resamples, n)`` for ``method='BCa'``).
vectorized : bool, optional
If `vectorized` is set ``False``, `statistic` will not be passed
keyword argument `axis` and is expected to calculate the statistic
only for 1D samples. If ``True``, `statistic` will be passed keyword
argument `axis` and is expected to calculate the statistic along `axis`
when passed an ND sample array. If ``None`` (default), `vectorized`
will be set ``True`` if ``axis`` is a parameter of `statistic`. Use of
a vectorized statistic typically reduces computation time.
paired : bool, default: ``False``
Whether the statistic treats corresponding elements of the samples
in `data` as paired.
axis : int, default: ``0``
The axis of the samples in `data` along which the `statistic` is
calculated.
confidence_level : float, default: ``0.95``
The confidence level of the confidence interval.
alternative : {'two-sided', 'less', 'greater'}, default: ``'two-sided'``
Choose ``'two-sided'`` (default) for a two-sided confidence interval,
``'less'`` for a one-sided confidence interval with the lower bound
at ``-np.inf``, and ``'greater'`` for a one-sided confidence interval
with the upper bound at ``np.inf``. The other bound of the one-sided
confidence intervals is the same as that of a two-sided confidence
interval with `confidence_level` twice as far from 1.0; e.g. the upper
bound of a 95% ``'less'`` confidence interval is the same as the upper
bound of a 90% ``'two-sided'`` confidence interval.
method : {'percentile', 'basic', 'bca'}, default: ``'BCa'``
Whether to return the 'percentile' bootstrap confidence interval
(``'percentile'``), the 'basic' (AKA 'reverse') bootstrap confidence
interval (``'basic'``), or the bias-corrected and accelerated bootstrap
confidence interval (``'BCa'``).
bootstrap_result : BootstrapResult, optional
Provide the result object returned by a previous call to `bootstrap`
to include the previous bootstrap distribution in the new bootstrap
distribution. This can be used, for example, to change
`confidence_level`, change `method`, or see the effect of performing
additional resampling without repeating computations.
random_state : {None, int, `numpy.random.Generator`,
`numpy.random.RandomState`}, optional
Pseudorandom number generator state used to generate resamples.
If `random_state` is ``None`` (or `np.random`), the
`numpy.random.RandomState` singleton is used.
If `random_state` is an int, a new ``RandomState`` instance is used,
seeded with `random_state`.
If `random_state` is already a ``Generator`` or ``RandomState``
instance then that instance is used.
Returns
-------
res : BootstrapResult
An object with attributes:
confidence_interval : ConfidenceInterval
The bootstrap confidence interval as an instance of
`collections.namedtuple` with attributes `low` and `high`.
bootstrap_distribution : ndarray
The bootstrap distribution, that is, the value of `statistic` for
each resample. The last dimension corresponds with the resamples
(e.g. ``res.bootstrap_distribution.shape[-1] == n_resamples``).
standard_error : float or ndarray
The bootstrap standard error, that is, the sample standard
deviation of the bootstrap distribution.
Warns
-----
`~scipy.stats.DegenerateDataWarning`
Generated when ``method='BCa'`` and the bootstrap distribution is
degenerate (e.g. all elements are identical).
Notes
-----
Elements of the confidence interval may be NaN for ``method='BCa'`` if
the bootstrap distribution is degenerate (e.g. all elements are identical).
In this case, consider using another `method` or inspecting `data` for
indications that other analysis may be more appropriate (e.g. all
observations are identical).
References
----------
.. [1] B. Efron and R. J. Tibshirani, An Introduction to the Bootstrap,
Chapman & Hall/CRC, Boca Raton, FL, USA (1993)
.. [2] Nathaniel E. Helwig, "Bootstrap Confidence Intervals",
http://users.stat.umn.edu/~helwig/notes/bootci-Notes.pdf
.. [3] Bootstrapping (statistics), Wikipedia,
https://en.wikipedia.org/wiki/Bootstrapping_%28statistics%29
Examples
--------
Suppose we have sampled data from an unknown distribution.
>>> import numpy as np
>>> rng = np.random.default_rng()
>>> from scipy.stats import norm
>>> dist = norm(loc=2, scale=4) # our "unknown" distribution
>>> data = dist.rvs(size=100, random_state=rng)
We are interested in the standard deviation of the distribution.
>>> std_true = dist.std() # the true value of the statistic
>>> print(std_true)
4.0
>>> std_sample = np.std(data) # the sample statistic
>>> print(std_sample)
3.9460644295563863
The bootstrap is used to approximate the variability we would expect if we
were to repeatedly sample from the unknown distribution and calculate the
statistic of the sample each time. It does this by repeatedly resampling
values *from the original sample* with replacement and calculating the
statistic of each resample. This results in a "bootstrap distribution" of
the statistic.
>>> import matplotlib.pyplot as plt
>>> from scipy.stats import bootstrap
>>> data = (data,) # samples must be in a sequence
>>> res = bootstrap(data, np.std, confidence_level=0.9,
... random_state=rng)
>>> fig, ax = plt.subplots()
>>> ax.hist(res.bootstrap_distribution, bins=25)
>>> ax.set_title('Bootstrap Distribution')
>>> ax.set_xlabel('statistic value')
>>> ax.set_ylabel('frequency')
>>> plt.show()
The standard error quantifies this variability. It is calculated as the
standard deviation of the bootstrap distribution.
>>> res.standard_error
0.24427002125829136
>>> res.standard_error == np.std(res.bootstrap_distribution, ddof=1)
True
The bootstrap distribution of the statistic is often approximately normal
with scale equal to the standard error.
>>> x = np.linspace(3, 5)
>>> pdf = norm.pdf(x, loc=std_sample, scale=res.standard_error)
>>> fig, ax = plt.subplots()
>>> ax.hist(res.bootstrap_distribution, bins=25, density=True)
>>> ax.plot(x, pdf)
>>> ax.set_title('Normal Approximation of the Bootstrap Distribution')
>>> ax.set_xlabel('statistic value')
>>> ax.set_ylabel('pdf')
>>> plt.show()
This suggests that we could construct a 90% confidence interval on the
statistic based on quantiles of this normal distribution.
>>> norm.interval(0.9, loc=std_sample, scale=res.standard_error)
(3.5442759991341726, 4.3478528599786)
Due to central limit theorem, this normal approximation is accurate for a
variety of statistics and distributions underlying the samples; however,
the approximation is not reliable in all cases. Because `bootstrap` is
designed to work with arbitrary underlying distributions and statistics,
it uses more advanced techniques to generate an accurate confidence
interval.
>>> print(res.confidence_interval)
ConfidenceInterval(low=3.57655333533867, high=4.382043696342881)
If we sample from the original distribution 1000 times and form a bootstrap
confidence interval for each sample, the confidence interval
contains the true value of the statistic approximately 90% of the time.
>>> n_trials = 1000
>>> ci_contains_true_std = 0
>>> for i in range(n_trials):
... data = (dist.rvs(size=100, random_state=rng),)
... ci = bootstrap(data, np.std, confidence_level=0.9, n_resamples=1000,
... random_state=rng).confidence_interval
... if ci[0] < std_true < ci[1]:
... ci_contains_true_std += 1
>>> print(ci_contains_true_std)
875
Rather than writing a loop, we can also determine the confidence intervals
for all 1000 samples at once.
>>> data = (dist.rvs(size=(n_trials, 100), random_state=rng),)
>>> res = bootstrap(data, np.std, axis=-1, confidence_level=0.9,
... n_resamples=1000, random_state=rng)
>>> ci_l, ci_u = res.confidence_interval
Here, `ci_l` and `ci_u` contain the confidence interval for each of the
``n_trials = 1000`` samples.
>>> print(ci_l[995:])
[3.77729695 3.75090233 3.45829131 3.34078217 3.48072829]
>>> print(ci_u[995:])
[4.88316666 4.86924034 4.32032996 4.2822427 4.59360598]
And again, approximately 90% contain the true value, ``std_true = 4``.
>>> print(np.sum((ci_l < std_true) & (std_true < ci_u)))
900
`bootstrap` can also be used to estimate confidence intervals of
multi-sample statistics, including those calculated by hypothesis
tests. `scipy.stats.mood` perform's Mood's test for equal scale parameters,
and it returns two outputs: a statistic, and a p-value. To get a
confidence interval for the test statistic, we first wrap
`scipy.stats.mood` in a function that accepts two sample arguments,
accepts an `axis` keyword argument, and returns only the statistic.
>>> from scipy.stats import mood
>>> def my_statistic(sample1, sample2, axis):
... statistic, _ = mood(sample1, sample2, axis=-1)
... return statistic
Here, we use the 'percentile' method with the default 95% confidence level.
>>> sample1 = norm.rvs(scale=1, size=100, random_state=rng)
>>> sample2 = norm.rvs(scale=2, size=100, random_state=rng)
>>> data = (sample1, sample2)
>>> res = bootstrap(data, my_statistic, method='basic', random_state=rng)
>>> print(mood(sample1, sample2)[0]) # element 0 is the statistic
-5.521109549096542
>>> print(res.confidence_interval)
ConfidenceInterval(low=-7.255994487314675, high=-4.016202624747605)
The bootstrap estimate of the standard error is also available.
>>> print(res.standard_error)
0.8344963846318795
Paired-sample statistics work, too. For example, consider the Pearson
correlation coefficient.
>>> from scipy.stats import pearsonr
>>> n = 100
>>> x = np.linspace(0, 10, n)
>>> y = x + rng.uniform(size=n)
>>> print(pearsonr(x, y)[0]) # element 0 is the statistic
0.9962357936065914
We wrap `pearsonr` so that it returns only the statistic.
>>> def my_statistic(x, y):
... return pearsonr(x, y)[0]
We call `bootstrap` using ``paired=True``.
Also, since ``my_statistic`` isn't vectorized to calculate the statistic
along a given axis, we pass in ``vectorized=False``.
>>> res = bootstrap((x, y), my_statistic, vectorized=False, paired=True,
... random_state=rng)
>>> print(res.confidence_interval)
ConfidenceInterval(low=0.9950085825848624, high=0.9971212407917498)
The result object can be passed back into `bootstrap` to perform additional
resampling:
>>> len(res.bootstrap_distribution)
9999
>>> res = bootstrap((x, y), my_statistic, vectorized=False, paired=True,
... n_resamples=1001, random_state=rng,
... bootstrap_result=res)
>>> len(res.bootstrap_distribution)
11000
or to change the confidence interval options:
>>> res2 = bootstrap((x, y), my_statistic, vectorized=False, paired=True,
... n_resamples=0, random_state=rng, bootstrap_result=res,
... method='percentile', confidence_level=0.9)
>>> np.testing.assert_equal(res2.bootstrap_distribution,
... res.bootstrap_distribution)
>>> res.confidence_interval
ConfidenceInterval(low=0.9950035351407804, high=0.9971170323404578)
without repeating computation of the original bootstrap distribution.
Nr r )rF rG r r r r� rJ r� )r rZ rY r< c �2 � t j | |d�� S )Nr )rL �qr )r rX )rL r� s r% �percentile_funz!bootstrap.<locals>.percentile_fun� s � ��=�=�1���3� 3r' �d r� r� r� )�ddofr )r� r� r� )r� r� r4 r5 rH rh r ri r� r` � full_like�infr� r �std)r" r$ rF r< r� r� r r� r� r� r� rG �argsrY r> r? r@ �resampled_datar- �resamplerZ �intervalr� �ci_l�ci_url s r% r r s- � �R
��y�*�f�d�)�;� �U� �!1�<� A�D�
��T�9�j�&�$�0@��+�u�f�.>��)�1�2�)�@�@�A� � �-�[�-�A�M�
�1�k�=�
1�
@���=�+�a�-�8� ���� ,�F�*�6�|�8D�F�H� � !� !�(� +� ,� ���9�n�>�2�>�?�
@� �.�.��2�6�K� *5� �)C�a�"�"�A�
%��'�'�
�
��� ��y�r��-8��G�GI��K��/���!�E�'�>�� 4� �+�x��{�3�� 7�D� �+�x��{�3�� 7�D�
��� �t�-�"�-� ��y�[�4�'��9��t�);�d���f� ��|�|�D�2�6�6�'�*�� � � !��|�|�D�"�&�&�)�� �/A�$��/M�2=�*,�&�&��1�2�*N� P� Pr' c �$ � t |� }||k7 r t d� �|dvr t d� �t |t � s|f}| f} |D ] } t | � r�t d� � t
|� t
| � k( s
d}
t |
� �t |� s t d� �|�!dt j |� j v }|s t |� } n|} t | |� } g } | D ]? }
t j |
� }
t j |
|d� }
| j |
� �A t |� }||k7 s|d k r t d
� �|�|}n t |� }||k7 s|d k r t d � �h d �}|j! � }||vrt d
|� �� �| || |||||fS )z(Input validation for `monte_carlo_test`.r� r� r� z0`rvs` must be callable or sequence of callables.z:If `rvs` is a sequence, `len(rvs)` must equal `len(data)`.z `statistic` must be callable.r r r �)`n_resamples` must be a positive integer.r� > r� r� r� �`alternative` must be in )r� r� �
isinstancer �callabler� r� r� r� r� r0 r r r� r+ rh r� )r" �rvsr$ r� rF r< r� r r� �rvs_ir� �statistic_vectorizedr� r- r� r� r� s r% �_monte_carlo_test_ivr� � s� � � �4�y�H� �x���5�6�6��,�,��K�L�L� �c�8� $��f���w��� P������ N�O� O�P� �s�8�s�4�y� �N����!�!� �I� ��7�8�8����w� 0� 0�� ;� F� F�F�
� � 3�I� >� � (� � �T�4� (�D��G�� ��� � �v�&�����V�X�r�2�����v� � �
�+�&�O��o�%� �A�)=��D�E�E� �}����u�:�� �H� ��A�
� � J�K� K�3�L� �#�#�%�K��,�&��4�\�N�C�D�D� �S�.�
�O� �k�8� -� -r' c �0 � e Zd ZU dZded< ded< ded< y)�MonteCarloTestResultat Result object returned by `scipy.stats.monte_carlo_test`.
Attributes
----------
statistic : float or ndarray
The observed test statistic of the sample.
pvalue : float or ndarray
The p-value for the given alternative.
null_distribution : ndarray
The values of the test statistic generated under the null
hypothesis.
r� r$ �pvaluer� �null_distributionNr� r( r' r% r� r� � � � � � "�!� � �!�!r' r� r- r" )r� rF r< r� r c
� ���� t | |||�|||� }|\ } }}}�}}}t j || ddi�� d } | D �
cg c] }
|
j d �� } }
|xs �} g }
t d�| � D ]O }t | �|z
� }t
|| � D ��cg c] \ }} |||f�� �� }}}|
j ||ddi�� �Q t j |
� }
|
j dgdg| j z z � }
�fd���fd����fd �}��|d
�} || |
| � }t j |dd� }t | ||
� S c c}
w c c}}w ) a� Perform a Monte Carlo hypothesis test.
`data` contains a sample or a sequence of one or more samples. `rvs`
specifies the distribution(s) of the sample(s) in `data` under the null
hypothesis. The value of `statistic` for the given `data` is compared
against a Monte Carlo null distribution: the value of the statistic for
each of `n_resamples` sets of samples generated using `rvs`. This gives
the p-value, the probability of observing such an extreme value of the
test statistic under the null hypothesis.
Parameters
----------
data : array-like or sequence of array-like
An array or sequence of arrays of observations.
rvs : callable or tuple of callables
A callable or sequence of callables that generates random variates
under the null hypothesis. Each element of `rvs` must be a callable
that accepts keyword argument ``size`` (e.g. ``rvs(size=(m, n))``) and
returns an N-d array sample of that shape. If `rvs` is a sequence, the
number of callables in `rvs` must match the number of samples in
`data`, i.e. ``len(rvs) == len(data)``. If `rvs` is a single callable,
`data` is treated as a single sample.
statistic : callable
Statistic for which the p-value of the hypothesis test is to be
calculated. `statistic` must be a callable that accepts a sample
(e.g. ``statistic(sample)``) or ``len(rvs)`` separate samples (e.g.
``statistic(samples1, sample2)`` if `rvs` contains two callables and
`data` contains two samples) and returns the resulting statistic.
If `vectorized` is set ``True``, `statistic` must also accept a keyword
argument `axis` and be vectorized to compute the statistic along the
provided `axis` of the samples in `data`.
vectorized : bool, optional
If `vectorized` is set ``False``, `statistic` will not be passed
keyword argument `axis` and is expected to calculate the statistic
only for 1D samples. If ``True``, `statistic` will be passed keyword
argument `axis` and is expected to calculate the statistic along `axis`
when passed ND sample arrays. If ``None`` (default), `vectorized`
will be set ``True`` if ``axis`` is a parameter of `statistic`. Use of
a vectorized statistic typically reduces computation time.
n_resamples : int, default: 9999
Number of samples drawn from each of the callables of `rvs`.
Equivalently, the number statistic values under the null hypothesis
used as the Monte Carlo null distribution.
batch : int, optional
The number of Monte Carlo samples to process in each call to
`statistic`. Memory usage is O(`batch`*``sample.size[axis]``). Default
is ``None``, in which case `batch` equals `n_resamples`.
alternative : {'two-sided', 'less', 'greater'}
The alternative hypothesis for which the p-value is calculated.
For each alternative, the p-value is defined as follows.
- ``'greater'`` : the percentage of the null distribution that is
greater than or equal to the observed value of the test statistic.
- ``'less'`` : the percentage of the null distribution that is
less than or equal to the observed value of the test statistic.
- ``'two-sided'`` : twice the smaller of the p-values above.
axis : int, default: 0
The axis of `data` (or each sample within `data`) over which to
calculate the statistic.
Returns
-------
res : MonteCarloTestResult
An object with attributes:
statistic : float or ndarray
The test statistic of the observed `data`.
pvalue : float or ndarray
The p-value for the given alternative.
null_distribution : ndarray
The values of the test statistic generated under the null
hypothesis.
References
----------
.. [1] B. Phipson and G. K. Smyth. "Permutation P-values Should Never Be
Zero: Calculating Exact P-values When Permutations Are Randomly Drawn."
Statistical Applications in Genetics and Molecular Biology 9.1 (2010).
Examples
--------
Suppose we wish to test whether a small sample has been drawn from a normal
distribution. We decide that we will use the skew of the sample as a
test statistic, and we will consider a p-value of 0.05 to be statistically
significant.
>>> import numpy as np
>>> from scipy import stats
>>> def statistic(x, axis):
... return stats.skew(x, axis)
After collecting our data, we calculate the observed value of the test
statistic.
>>> rng = np.random.default_rng()
>>> x = stats.skewnorm.rvs(a=1, size=50, random_state=rng)
>>> statistic(x, axis=0)
0.12457412450240658
To determine the probability of observing such an extreme value of the
skewness by chance if the sample were drawn from the normal distribution,
we can perform a Monte Carlo hypothesis test. The test will draw many
samples at random from their normal distribution, calculate the skewness
of each sample, and compare our original skewness against this
distribution to determine an approximate p-value.
>>> from scipy.stats import monte_carlo_test
>>> # because our statistic is vectorized, we pass `vectorized=True`
>>> rvs = lambda size: stats.norm.rvs(size=size, random_state=rng)
>>> res = monte_carlo_test(x, rvs, statistic, vectorized=True)
>>> print(res.statistic)
0.12457412450240658
>>> print(res.pvalue)
0.7012
The probability of obtaining a test statistic less than or equal to the
observed value under the null hypothesis is ~70%. This is greater than
our chosen threshold of 5%, so we cannot consider this to be significant
evidence against the null hypothesis.
Note that this p-value essentially matches that of
`scipy.stats.skewtest`, which relies on an asymptotic distribution of a
test statistic based on the sample skewness.
>>> stats.skewtest(x).pvalue
0.6892046027110614
This asymptotic approximation is not valid for small sample sizes, but
`monte_carlo_test` can be used with samples of any size.
>>> x = stats.skewnorm.rvs(a=1, size=7, random_state=rng)
>>> # stats.skewtest(x) would produce an error due to small sample
>>> res = monte_carlo_test(x, rvs, statistic, vectorized=True)
The Monte Carlo distribution of the test statistic is provided for
further investigation.
>>> import matplotlib.pyplot as plt
>>> fig, ax = plt.subplots()
>>> ax.hist(res.null_distribution, bins=50)
>>> ax.set_title("Monte Carlo distribution of test statistic")
>>> ax.set_xlabel("Value of Statistic")
>>> ax.set_ylabel("Frequency")
>>> plt.show()
r r r( r �r� r c �H �� | |k }|j d�� dz �dz z }|S �Nr r r �rK �r� �observed�cmps�pvaluesrF s �r% r� z monte_carlo_test.<locals>.less� �0 �� � �H�,���8�8��8�#�a�'�K�!�O�<���r' c �H �� | |k\ }|j d�� dz �dz z }|S r� r� r� s �r% r� z!monte_carlo_test.<locals>.greater� r� r' c �^ �� �| |� } �| |� }t j ||� dz }|S �NrJ �r �minimum�r� r� � pvalues_less�pvalues_greaterr� r� r� s ��r% � two_sidedz#monte_carlo_test.<locals>.two_sided� �7 �� �� -�x�8� �!�"3�X�>���*�*�\�?�;�a�?���r' �r� r� r� )
r� r re r) r4 r5 rk rh ri r; �ndim�clipr� )r" r� r$ r� rF r< r� r r� r� r- �n_observationsr> r� r? r@ r� �n_observations_irC r� �comparer� r� r� s ` @@r% r r � s� �� �r ��c�9�j� +�U�K�� G�D� /3�,�T�3� �:��%��d� �z�z�)�T�3��3�4�R�8�H�59�:�6�f�l�l�2�&�:�N�:��(�[�M���
�1�k�=�
1� A���=�+��/�:� �47��^�4L�N�0�� 0� � �/?� @�A� N� � N�� � ��I�!?�B�!?�@� A�
���'8�9��)�1�1�2�$�!��X�]�]�9J�2J�K���
�
� �!� %�'�G� #�g�k�"�#4�h�?�G��g�g�g�q�!�$�G� ��'�3D� E�E��E ;��
Ns �E �Ec �0 � e Zd ZU dZded< ded< ded< y)�PermutationTestResultar Result object returned by `scipy.stats.permutation_test`.
Attributes
----------
statistic : float or ndarray
The observed test statistic of the data.
pvalue : float or ndarray
The p-value for the given alternative.
null_distribution : ndarray
The values of the test statistic generated under the null
hypothesis.
r� r$ r� r� r� Nr� r( r' r% r� r� � r� r' r� c # �" ��K � d� ���fd��t t t j | � � � } �|| dd � D ]E }t j |D �cg c]
}t |� �� c}� j
t � }|�� �G yc c}w �w)zu
Generate all partitions of indices of groups of given sizes, concatenated
`ns` is an iterable of ints.
c 3 �Z K � t | |� D ] }t |� }| |z
}||g�� � y �wr )r �set)r! r= �c�x0�x1s r% �all_partitionsz4_all_partitions_concatenated.<locals>.all_partitions� s7 � �� � �a��#� �A��Q��B��R��B��r�(�N� �s �)+c 3 � �K � t |� dk( r| g�� y �| |d � D ] } �|d |dd � D ] }|dd |z �� � �" y �w)Nr r )r� )r! �nsr� �dr� �all_partitions_ns ��r% r� z6_all_partitions_concatenated.<locals>.all_partitions_n� sf �� �� � �r�7�a�<��#�I� � ��2�a�5�)� !�A�%�a��d�B�q�r�F�3�
!����!�f�q�j� �
!� !�s �AAN)r� r4 r rK ri �list�astyper� )r� r! � partitioning� partition�xr� r� s @@r% � _all_partitions_concatenatedr � s� �� �� � � !� �E�"�&�&��*� � �A�(��B�q�E�2� � � �N�N�-9� ;� )� !��O� ;�
<�<B�F�3�K�
����� ;�s �AB�B
�)&Bc # �
K � t | � }|dk r t d� �t t |� |� D ��cg c] \ }}|�� }}}|r/|�� t t |� |� D ��cg c] \ }}|�� }}}|r�.yyc c}}w c c}}w �w)z<A generator that yields batches of elements from an iterabler z`batch` must be positive.N)�iterr� rk r4 )�iterabler< �iteratorrB �itemr! s r% �_batch_generatorr � s} � �� ��H�~�H� ��z��4�5�5� ��u� �x� 8�9�'�!�T��9�A�9�
���!$�U�5�\�8�!<� =�g�a��T� =�� =� �� :��
>�s" �4B� A7�!B�# A=�/B�5Bc �| � ����� t �� � �t �d� r��� ��fd�} |� S ��� ��fd�} |� S )N�permutedc 3 �� �K � t j �� } t j | ��df� } t d��� D ]+ }t ��|z
� }�j | d�� }|d | �� �- y �w)Nr r r r )r r9 �tiler4 r5 r ) r\ r? r@ �permuted_indicesr< � n_obs_sample�n_permutationsr� rG s �����r% �batched_perm_generatorz:_pairings_permutations_gen.<locals>.batched_perm_generator� sx �� �� ��i�i� �-�G��g�g�g��y�!�'<�=�G��1�n�e�4�
6�� "�5�.��*:� ;� �#/�#8�#8��r�#8�#J� �&�}� �5�5�
6�s �A+A.c 3 � �K � t d��� D ]D } t ��| z
� }|��f}�j |�� }t j |d�� d | �� �F y �w)Nr r� r r )r4 r5 �randomr �argsort) r? r@ r� r r< r r r� rG s �����r% r z:_pairings_permutations_gen.<locals>.batched_perm_generator� sg �� �� ��1�n�e�4�
<�� "�5�.��*:� ;� �$�i� �>�� �'�'�T�'�2���j�j���,�]�l�;�;�
<�s �AA)r5 r� )r r� r r< rG r s ````` r% �_pairings_permutations_genr � sE �� �
��~� &�E��|�Z�(� 6� 6� "� #�#� <� <� "� #�#r' c �� ��� t | � }| D �cg c] }|j d �� }}t j |� }|d �t j t |dz
dd� D � cg c] } t
|| || dz
� �� c} � }
||
k\ rd} |
}t |� } nd} ��fd�t |� D � } |xs t |� }g }
t j | d�� } t | |�� D ]g }t j |� }| d |f }t j |d
d� }t j ||d d d�� }|
j ||d di�� �i t j |
d�� }
|
|| fS c c}w c c} w )
zC
Calculate null distribution for independent sample tests.
r r r TFc 3 �@ �K � | ] }�j �� �� � y �wr )� permutation)�.0rB �n_obsrG s ��r% � <genexpr>z'_calculate_null_both.<locals>.<genexpr> s$ �� �� � :� � '�2�2�5�9� :�s � r �r< .rb Nr )r� r) r r* �prodr4 r r r� ri r �arrayr+ r rh )r" r$ r r< rG r� r- �n_obs_i�n_obs_icrB �n_max�
exact_test�perm_generatorr� r\ �
data_batchr s ` @r% �_calculate_null_bothr% s� �� �
�D� �I� /3�3�F�v�|�|�B� �3�G�3��y�y��!�H� �R�L�E� �G�G� #�I�a�K��B� 7�9�� �(�1�+�x��!��}�5� 9�
:�E� �� ��
� ��5�g�>���
�
:�#(� �#8�:��
� (�S� �(�E��� �>�>�$�R� (�D�#�N�%�@� B���(�(�7�#�� �#�w�,�'�
�
�[�[��R��3�
��X�X�j�(�3�B�-�b�A�
�� � ��J�!@�R�!@�A� B� ���'8�q�A�� �n�j� 8�8��a 4��9s �E/�% E4c �p �� t | � }| d j d �t �� |z }||k\ r;d}|}|xs t |� }t �fd�t |� D � � }t
||�� } n d}|xs t |� }||�||f}
t |
� } g } | D ]� } t j | � } t j | dd� } dg|z }
t |� D ]/ }| | d | | f |
|< t j |
| d
d� |
|< �1 | j ||
d di�� �� t j | d� � } | ||fS )
z<
Calculate null distribution for association tests.
r r Tc 3 �F �K � | ] }t t �� � �� � y �wr )r r4 )r rB r s �r% r z+_calculate_null_pairings.<locals>.<genexpr>N s# �� �� � #>�'(� $0��l�0C�#D� #>�s � !r Fr N.rb r r )r� r) r
r� r r4 r r r r �swapaxesr+ rh ri )r" r$ r r< rG r� r! r" r# r r� r� r\ r$ rB r s @r% �_calculate_null_pairingsr) < sx �� �
�D� �I� ��7�=�=��$�L� �l� #�Y� .�E� �� ��
� ���,��^�,�� � #>�,1�)�,<�#>� ?��!1�.��!N� ��
��,��^�,�� �y�,��|�K��!;�T�!B� ���)� B���(�(�7�#�� �+�+�g�q�!�,�� �V�I�%�
��y�!� >�A� ��G�C����O� 4�J�q�M� �K�K�
�1�
�r�1� =�J�q�M� >� � � ��J�!@�R�!@�A�'B�( ���'8�q�A�� �n�j� 8�8r' c � ��� t | � ��dk( r | d | d g} t j | dd� } ��fd�}t | ||||� S )z>
Calculate null distribution for paired-sample tests.
r r r c �V �� t j |dd� }�dk( r|dd } �|d| i�S )Nr r r r )r r( )r r" r� r$ s ��r% �statistic_wrappedz2_calculate_null_samples.<locals>.statistic_wrapped� s7 �� ��{�{�4��B�'�� ��>���!�9�D��$�*�T�*�*r' )r� r r( r) )r" r$ r r< rG r, r� s ` @r% �_calculate_null_samplesr- u s_ �� �
�D� �I�
�A�~��Q��$�q�'��"�� �;�;�t�Q�� #�D�+� $�D�*;�^�$)�<� 9� 9r' c � � t |� } || k7 r t d� �h d�}
|j � }||
vrt d|
� d�� �|dvr t d� �|�!dt j |� j
v }|s t
|� }d} t | � d k r|d
k( r t | � �t | |� } g } | D ]\ }
t j |
� }
|
j | d k r t d � �t j |
| d
� }
| j |
� �^ t j |� s t |� nt j }||k7 s|dk r t d� �|�|}n t |� }||k7 s|dk r t d� �h d�}|j � }||vrt d|� �� �t# |� }| ||||||| |f S # t $ r t | � �w xY w)z(Input validation for `permutation_test`.r� > ro �pairings� independentz `permutation_type` must be in �.r� r� r z6`data` must be a tuple containing at least two samplesrJ r0 r r� r r r� r� > r� r� r� r� )r� r� r� r� r� r� r0 r� r� r r r� r) r+ rh �isinfr� r )r" r$ �permutation_typer� rF r< r� r rG r� �permutation_typesr� r� r- r� r� r� s r% �_permutation_test_ivr5 � s � � �4�y�H� �x���5�6�6�>��'�-�-�/��� 0�0��9�:K�9L�A�N�O�O��,�,��K�L�L����w� 0� 0�� ;� F� F�F�
� �(��3� �F�G�!� �t�9�q�=� -� � >� �W�%� %� �T�4� (�D��G�� ��� � �v�&�� �<�<�� �� "� � :� ;�
;����V�X�r�2�����v� �
� 02�x�x� �/D�s�;�'� �F�F� ��o�%� �A�)=��D�E�E� �}����u�:�� �H� ��A�
� � J�K� K�3�L� �#�#�%�K��,�&��4�\�N�C�D�D�%�l�3�L� �Y� 0�*�o� �k�8�\� ;� ;��A � !��� � �!�s � F8 �8G
r0 )r3 r� rF r< r� r rG c � ������ t | |||�||||� } | \ } }}}�}}}} || ddi�}
t t t d�} | |�||f} | | }
|
| � \ }�}|rdnd�d}t j
|t j ||
z � � ����fd�����fd����fd �}��|d
�} || ||
� }t j |dd� }t |
||� S ) a�Q
Performs a permutation test of a given statistic on provided data.
For independent sample statistics, the null hypothesis is that the data are
randomly sampled from the same distribution.
For paired sample statistics, two null hypothesis can be tested:
that the data are paired at random or that the data are assigned to samples
at random.
Parameters
----------
data : iterable of array-like
Contains the samples, each of which is an array of observations.
Dimensions of sample arrays must be compatible for broadcasting except
along `axis`.
statistic : callable
Statistic for which the p-value of the hypothesis test is to be
calculated. `statistic` must be a callable that accepts samples
as separate arguments (e.g. ``statistic(*data)``) and returns the
resulting statistic.
If `vectorized` is set ``True``, `statistic` must also accept a keyword
argument `axis` and be vectorized to compute the statistic along the
provided `axis` of the sample arrays.
permutation_type : {'independent', 'samples', 'pairings'}, optional
The type of permutations to be performed, in accordance with the
null hypothesis. The first two permutation types are for paired sample
statistics, in which all samples contain the same number of
observations and observations with corresponding indices along `axis`
are considered to be paired; the third is for independent sample
statistics.
- ``'samples'`` : observations are assigned to different samples
but remain paired with the same observations from other samples.
This permutation type is appropriate for paired sample hypothesis
tests such as the Wilcoxon signed-rank test and the paired t-test.
- ``'pairings'`` : observations are paired with different observations,
but they remain within the same sample. This permutation type is
appropriate for association/correlation tests with statistics such
as Spearman's :math:`\rho`, Kendall's :math:`\tau`, and Pearson's
:math:`r`.
- ``'independent'`` (default) : observations are assigned to different
samples. Samples may contain different numbers of observations. This
permutation type is appropriate for independent sample hypothesis
tests such as the Mann-Whitney :math:`U` test and the independent
sample t-test.
Please see the Notes section below for more detailed descriptions
of the permutation types.
vectorized : bool, optional
If `vectorized` is set ``False``, `statistic` will not be passed
keyword argument `axis` and is expected to calculate the statistic
only for 1D samples. If ``True``, `statistic` will be passed keyword
argument `axis` and is expected to calculate the statistic along `axis`
when passed an ND sample array. If ``None`` (default), `vectorized`
will be set ``True`` if ``axis`` is a parameter of `statistic`. Use
of a vectorized statistic typically reduces computation time.
n_resamples : int or np.inf, default: 9999
Number of random permutations (resamples) used to approximate the null
distribution. If greater than or equal to the number of distinct
permutations, the exact null distribution will be computed.
Note that the number of distinct permutations grows very rapidly with
the sizes of samples, so exact tests are feasible only for very small
data sets.
batch : int, optional
The number of permutations to process in each call to `statistic`.
Memory usage is O(`batch`*``n``), where ``n`` is the total size
of all samples, regardless of the value of `vectorized`. Default is
``None``, in which case ``batch`` is the number of permutations.
alternative : {'two-sided', 'less', 'greater'}, optional
The alternative hypothesis for which the p-value is calculated.
For each alternative, the p-value is defined for exact tests as
follows.
- ``'greater'`` : the percentage of the null distribution that is
greater than or equal to the observed value of the test statistic.
- ``'less'`` : the percentage of the null distribution that is
less than or equal to the observed value of the test statistic.
- ``'two-sided'`` (default) : twice the smaller of the p-values above.
Note that p-values for randomized tests are calculated according to the
conservative (over-estimated) approximation suggested in [2]_ and [3]_
rather than the unbiased estimator suggested in [4]_. That is, when
calculating the proportion of the randomized null distribution that is
as extreme as the observed value of the test statistic, the values in
the numerator and denominator are both increased by one. An
interpretation of this adjustment is that the observed value of the
test statistic is always included as an element of the randomized
null distribution.
The convention used for two-sided p-values is not universal;
the observed test statistic and null distribution are returned in
case a different definition is preferred.
axis : int, default: 0
The axis of the (broadcasted) samples over which to calculate the
statistic. If samples have a different number of dimensions,
singleton dimensions are prepended to samples with fewer dimensions
before `axis` is considered.
random_state : {None, int, `numpy.random.Generator`,
`numpy.random.RandomState`}, optional
Pseudorandom number generator state used to generate permutations.
If `random_state` is ``None`` (default), the
`numpy.random.RandomState` singleton is used.
If `random_state` is an int, a new ``RandomState`` instance is used,
seeded with `random_state`.
If `random_state` is already a ``Generator`` or ``RandomState``
instance then that instance is used.
Returns
-------
res : PermutationTestResult
An object with attributes:
statistic : float or ndarray
The observed test statistic of the data.
pvalue : float or ndarray
The p-value for the given alternative.
null_distribution : ndarray
The values of the test statistic generated under the null
hypothesis.
Notes
-----
The three types of permutation tests supported by this function are
described below.
**Unpaired statistics** (``permutation_type='independent'``):
The null hypothesis associated with this permutation type is that all
observations are sampled from the same underlying distribution and that
they have been assigned to one of the samples at random.
Suppose ``data`` contains two samples; e.g. ``a, b = data``.
When ``1 < n_resamples < binom(n, k)``, where
* ``k`` is the number of observations in ``a``,
* ``n`` is the total number of observations in ``a`` and ``b``, and
* ``binom(n, k)`` is the binomial coefficient (``n`` choose ``k``),
the data are pooled (concatenated), randomly assigned to either the first
or second sample, and the statistic is calculated. This process is
performed repeatedly, `permutation` times, generating a distribution of the
statistic under the null hypothesis. The statistic of the original
data is compared to this distribution to determine the p-value.
When ``n_resamples >= binom(n, k)``, an exact test is performed: the data
are *partitioned* between the samples in each distinct way exactly once,
and the exact null distribution is formed.
Note that for a given partitioning of the data between the samples,
only one ordering/permutation of the data *within* each sample is
considered. For statistics that do not depend on the order of the data
within samples, this dramatically reduces computational cost without
affecting the shape of the null distribution (because the frequency/count
of each value is affected by the same factor).
For ``a = [a1, a2, a3, a4]`` and ``b = [b1, b2, b3]``, an example of this
permutation type is ``x = [b3, a1, a2, b2]`` and ``y = [a4, b1, a3]``.
Because only one ordering/permutation of the data *within* each sample
is considered in an exact test, a resampling like ``x = [b3, a1, b2, a2]``
and ``y = [a4, a3, b1]`` would *not* be considered distinct from the
example above.
``permutation_type='independent'`` does not support one-sample statistics,
but it can be applied to statistics with more than two samples. In this
case, if ``n`` is an array of the number of observations within each
sample, the number of distinct partitions is::
np.prod([binom(sum(n[i:]), sum(n[i+1:])) for i in range(len(n)-1)])
**Paired statistics, permute pairings** (``permutation_type='pairings'``):
The null hypothesis associated with this permutation type is that
observations within each sample are drawn from the same underlying
distribution and that pairings with elements of other samples are
assigned at random.
Suppose ``data`` contains only one sample; e.g. ``a, = data``, and we
wish to consider all possible pairings of elements of ``a`` with elements
of a second sample, ``b``. Let ``n`` be the number of observations in
``a``, which must also equal the number of observations in ``b``.
When ``1 < n_resamples < factorial(n)``, the elements of ``a`` are
randomly permuted. The user-supplied statistic accepts one data argument,
say ``a_perm``, and calculates the statistic considering ``a_perm`` and
``b``. This process is performed repeatedly, `permutation` times,
generating a distribution of the statistic under the null hypothesis.
The statistic of the original data is compared to this distribution to
determine the p-value.
When ``n_resamples >= factorial(n)``, an exact test is performed:
``a`` is permuted in each distinct way exactly once. Therefore, the
`statistic` is computed for each unique pairing of samples between ``a``
and ``b`` exactly once.
For ``a = [a1, a2, a3]`` and ``b = [b1, b2, b3]``, an example of this
permutation type is ``a_perm = [a3, a1, a2]`` while ``b`` is left
in its original order.
``permutation_type='pairings'`` supports ``data`` containing any number
of samples, each of which must contain the same number of observations.
All samples provided in ``data`` are permuted *independently*. Therefore,
if ``m`` is the number of samples and ``n`` is the number of observations
within each sample, then the number of permutations in an exact test is::
factorial(n)**m
Note that if a two-sample statistic, for example, does not inherently
depend on the order in which observations are provided - only on the
*pairings* of observations - then only one of the two samples should be
provided in ``data``. This dramatically reduces computational cost without
affecting the shape of the null distribution (because the frequency/count
of each value is affected by the same factor).
**Paired statistics, permute samples** (``permutation_type='samples'``):
The null hypothesis associated with this permutation type is that
observations within each pair are drawn from the same underlying
distribution and that the sample to which they are assigned is random.
Suppose ``data`` contains two samples; e.g. ``a, b = data``.
Let ``n`` be the number of observations in ``a``, which must also equal
the number of observations in ``b``.
When ``1 < n_resamples < 2**n``, the elements of ``a`` are ``b`` are
randomly swapped between samples (maintaining their pairings) and the
statistic is calculated. This process is performed repeatedly,
`permutation` times, generating a distribution of the statistic under the
null hypothesis. The statistic of the original data is compared to this
distribution to determine the p-value.
When ``n_resamples >= 2**n``, an exact test is performed: the observations
are assigned to the two samples in each distinct way (while maintaining
pairings) exactly once.
For ``a = [a1, a2, a3]`` and ``b = [b1, b2, b3]``, an example of this
permutation type is ``x = [b1, a2, b3]`` and ``y = [a1, b2, a3]``.
``permutation_type='samples'`` supports ``data`` containing any number
of samples, each of which must contain the same number of observations.
If ``data`` contains more than one sample, paired observations within
``data`` are exchanged between samples *independently*. Therefore, if ``m``
is the number of samples and ``n`` is the number of observations within
each sample, then the number of permutations in an exact test is::
factorial(m)**n
Several paired-sample statistical tests, such as the Wilcoxon signed rank
test and paired-sample t-test, can be performed considering only the
*difference* between two paired elements. Accordingly, if ``data`` contains
only one sample, then the null distribution is formed by independently
changing the *sign* of each observation.
.. warning::
The p-value is calculated by counting the elements of the null
distribution that are as extreme or more extreme than the observed
value of the statistic. Due to the use of finite precision arithmetic,
some statistic functions return numerically distinct values when the
theoretical values would be exactly equal. In some cases, this could
lead to a large error in the calculated p-value. `permutation_test`
guards against this by considering elements in the null distribution
that are "close" (within a factor of ``1+1e-14``) to the observed
value of the test statistic as equal to the observed value of the
test statistic. However, the user is advised to inspect the null
distribution to assess whether this method of comparison is
appropriate, and if not, calculate the p-value manually. See example
below.
References
----------
.. [1] R. A. Fisher. The Design of Experiments, 6th Ed (1951).
.. [2] B. Phipson and G. K. Smyth. "Permutation P-values Should Never Be
Zero: Calculating Exact P-values When Permutations Are Randomly Drawn."
Statistical Applications in Genetics and Molecular Biology 9.1 (2010).
.. [3] M. D. Ernst. "Permutation Methods: A Basis for Exact Inference".
Statistical Science (2004).
.. [4] B. Efron and R. J. Tibshirani. An Introduction to the Bootstrap
(1993).
Examples
--------
Suppose we wish to test whether two samples are drawn from the same
distribution. Assume that the underlying distributions are unknown to us,
and that before observing the data, we hypothesized that the mean of the
first sample would be less than that of the second sample. We decide that
we will use the difference between the sample means as a test statistic,
and we will consider a p-value of 0.05 to be statistically significant.
For efficiency, we write the function defining the test statistic in a
vectorized fashion: the samples ``x`` and ``y`` can be ND arrays, and the
statistic will be calculated for each axis-slice along `axis`.
>>> import numpy as np
>>> def statistic(x, y, axis):
... return np.mean(x, axis=axis) - np.mean(y, axis=axis)
After collecting our data, we calculate the observed value of the test
statistic.
>>> from scipy.stats import norm
>>> rng = np.random.default_rng()
>>> x = norm.rvs(size=5, random_state=rng)
>>> y = norm.rvs(size=6, loc = 3, random_state=rng)
>>> statistic(x, y, 0)
-3.5411688580987266
Indeed, the test statistic is negative, suggesting that the true mean of
the distribution underlying ``x`` is less than that of the distribution
underlying ``y``. To determine the probability of this occuring by chance
if the two samples were drawn from the same distribution, we perform
a permutation test.
>>> from scipy.stats import permutation_test
>>> # because our statistic is vectorized, we pass `vectorized=True`
>>> # `n_resamples=np.inf` indicates that an exact test is to be performed
>>> res = permutation_test((x, y), statistic, vectorized=True,
... n_resamples=np.inf, alternative='less')
>>> print(res.statistic)
-3.5411688580987266
>>> print(res.pvalue)
0.004329004329004329
The probability of obtaining a test statistic less than or equal to the
observed value under the null hypothesis is 0.4329%. This is less than our
chosen threshold of 5%, so we consider this to be significant evidence
against the null hypothesis in favor of the alternative.
Because the size of the samples above was small, `permutation_test` could
perform an exact test. For larger samples, we resort to a randomized
permutation test.
>>> x = norm.rvs(size=100, random_state=rng)
>>> y = norm.rvs(size=120, loc=0.3, random_state=rng)
>>> res = permutation_test((x, y), statistic, n_resamples=100000,
... vectorized=True, alternative='less',
... random_state=rng)
>>> print(res.statistic)
-0.5230459671240913
>>> print(res.pvalue)
0.00016999830001699983
The approximate probability of obtaining a test statistic less than or
equal to the observed value under the null hypothesis is 0.0225%. This is
again less than our chosen threshold of 5%, so again we have significant
evidence to reject the null hypothesis in favor of the alternative.
For large samples and number of permutations, the result is comparable to
that of the corresponding asymptotic test, the independent sample t-test.
>>> from scipy.stats import ttest_ind
>>> res_asymptotic = ttest_ind(x, y, alternative='less')
>>> print(res_asymptotic.pvalue)
0.00012688101537979522
The permutation distribution of the test statistic is provided for
further investigation.
>>> import matplotlib.pyplot as plt
>>> plt.hist(res.null_distribution, bins=50)
>>> plt.title("Permutation distribution of test statistic")
>>> plt.xlabel("Value of Statistic")
>>> plt.ylabel("Frequency")
>>> plt.show()
Inspection of the null distribution is essential if the statistic suffers
from inaccuracy due to limited machine precision. Consider the following
case:
>>> from scipy.stats import pearsonr
>>> x = [1, 2, 4, 3]
>>> y = [2, 4, 6, 8]
>>> def statistic(x, y):
... return pearsonr(x, y).statistic
>>> res = permutation_test((x, y), statistic, vectorized=False,
... permutation_type='pairings',
... alternative='greater')
>>> r, pvalue, null = res.statistic, res.pvalue, res.null_distribution
In this case, some elements of the null distribution differ from the
observed value of the correlation coefficient ``r`` due to numerical noise.
We manually inspect the elements of the null distribution that are nearly
the same as the observed value of the test statistic.
>>> r
0.8
>>> unique = np.unique(null)
>>> unique
array([-1. , -0.8, -0.8, -0.6, -0.4, -0.2, -0.2, 0. , 0.2, 0.2, 0.4,
0.6, 0.8, 0.8, 1. ]) # may vary
>>> unique[np.isclose(r, unique)].tolist()
[0.7999999999999999, 0.8]
If `permutation_test` were to perform the comparison naively, the
elements of the null distribution with value ``0.7999999999999999`` would
not be considered as extreme or more extreme as the observed value of the
statistic, so the calculated p-value would be too small.
>>> incorrect_pvalue = np.count_nonzero(null >= r) / len(null)
>>> incorrect_pvalue
0.1111111111111111 # may vary
Instead, `permutation_test` treats elements of the null distribution that
are within ``max(1e-14, abs(r)*1e-14)`` of the observed value of the
statistic ``r`` to be equal to ``r``.
>>> correct_pvalue = np.count_nonzero(null >= r - 1e-14) / len(null)
>>> correct_pvalue
0.16666666666666666
>>> res.pvalue == correct_pvalue
True
This method of comparison is expected to be accurate in most practical
situations, but the user is advised to assess this by inspecting the
elements of the null distribution that are close to the observed value
of the statistic. Also, consider the use of statistics that can be
calculated using exact arithmetic (e.g. integer statistics).
r r )r/ ro r0 r r g�+����=c �N �� | |�z k }|j d�� �z ��z z }|S �Nr r r� �r� r� r� r� �
adjustment�gammarF s ���r% r� z permutation_test.<locals>.less� �6 �� � �H�u�$4�4���8�8��8�#�j�0�[�:�5M�N���r' c �N �� | |�z
k\ }|j d�� �z ��z z }|S r8 r� r9 s ���r% r� z!permutation_test.<locals>.greater� r< r' c �^ �� �| |� } �| |� }t j ||� dz }|S r� r� r� s ��r% r� z#permutation_test.<locals>.two_sided� r� r' r� ) r5 r) r- r% r �maximum�absr� r� )r" r$ r3 r� rF r< r� r rG r� r� �null_calculators�null_calculator_args�calculate_nullr� r" �epsr� r� r� r: r; r� r� s ` @@@@r% r r � s! �� �R
��i�1A�:� +�U�K�� ,� .�D� )-�&�T�9�&�
�K���$� ��$�(�R�(�H�$<�#:�';�=�� !�)�[� !�<� 1��%�&6�7�N��,�-� /��{�J� !��a�J� �C� �J�J�s�B�F�F�3��>� 2� 3�E��
�
� �!� %�'�G� #�g�k�"�#4�h�?�G��g�g�g�q�!�$�G� ��7�4E� F�Fr' c �. � e Zd ZU dZdZded< dZded< y)�ResamplingMethoda� Configuration information for a statistical resampling method.
Instances of this class can be passed into the `method` parameter of some
hypothesis test functions to perform a resampling or Monte Carlo version
of the hypothesis test.
Attributes
----------
n_resamples : int
The number of resamples to perform or Monte Carlo samples to draw.
batch : int, optional
The number of resamples to process in each vectorized call to
the statistic. Batch sizes >>1 tend to be faster when the statistic
is vectorized, but memory usage scales linearly with the batch size.
Default is ``None``, which processes all resamples in a single batch.
r� r� rF Nr< )r� r� r� r� rF r� r< r( r' r% rF rF � s � �� �K����E�3�r' rF c �& � e Zd ZU dZdZded< d� Zy)�MonteCarloMethodaI Configuration information for a Monte Carlo hypothesis test.
Instances of this class can be passed into the `method` parameter of some
hypothesis test functions to perform a Monte Carlo version of the
hypothesis tests.
Attributes
----------
n_resamples : int, optional
The number of Monte Carlo samples to draw. Default is 9999.
batch : int, optional
The number of Monte Carlo samples to process in each vectorized call to
the statistic. Batch sizes >>1 tend to be faster when the statistic
is vectorized, but memory usage scales linearly with the batch size.
Default is ``None``, which processes all samples in a single batch.
rvs : callable or tuple of callables, optional
A callable or sequence of callables that generates random variates
under the null hypothesis. Each element of `rvs` must be a callable
that accepts keyword argument ``size`` (e.g. ``rvs(size=(m, n))``) and
returns an N-d array sample of that shape. If `rvs` is a sequence, the
number of callables in `rvs` must match the number of samples passed
to the hypothesis test in which the `MonteCarloMethod` is used. Default
is ``None``, in which case the hypothesis test function chooses values
to match the standard version of the hypothesis test. For example,
the null hypothesis of `scipy.stats.pearsonr` is typically that the
samples are drawn from the standard normal distribution, so
``rvs = (rng.normal, rng.normal)`` where
``rng = np.random.default_rng()``.
N�objectr� c �Z � t | j | j | j �� S )N)rF r< r� )�dictrF r< r� ��selfs r% �_asdictzMonteCarloMethod._asdict� s$ � ��� 0� 0��
�
� ���"� "r' )r� r� r� r� r� r� rN r( r' r% rH rH � s � � �: �C���"r' rH c �& � e Zd ZU dZdZded< d� Zy)�PermutationMethoda� Configuration information for a permutation hypothesis test.
Instances of this class can be passed into the `method` parameter of some
hypothesis test functions to perform a permutation version of the
hypothesis tests.
Attributes
----------
n_resamples : int, optional
The number of resamples to perform. Default is 9999.
batch : int, optional
The number of resamples to process in each vectorized call to
the statistic. Batch sizes >>1 tend to be faster when the statistic
is vectorized, but memory usage scales linearly with the batch size.
Default is ``None``, which processes all resamples in a single batch.
random_state : {None, int, `numpy.random.Generator`,
`numpy.random.RandomState`}, optional
Pseudorandom number generator state used to generate resamples.
If `random_state` is already a ``Generator`` or ``RandomState``
instance, then that instance is used.
If `random_state` is an int, a new ``RandomState`` instance is used,
seeded with `random_state`.
If `random_state` is ``None`` (default), the
`numpy.random.RandomState` singleton is used.
NrI rG c �Z � t | j | j | j �� S )N)rF r< rG )rK rF r< rG rL s r% rN zPermutationMethod._asdict
s& � ��� 0� 0��
�
�!%�!2�!2�4� 4r' )r� r� r� r� rG r� rN r( r' r% rP rP � s � ��6 �L�&� �4r' rP c �4 � e Zd ZU dZdZded< dZded< d� Zy) �BootstrapMethoda� Configuration information for a bootstrap confidence interval.
Instances of this class can be passed into the `method` parameter of some
confidence interval methods to generate a bootstrap confidence interval.
Attributes
----------
n_resamples : int, optional
The number of resamples to perform. Default is 9999.
batch : int, optional
The number of resamples to process in each vectorized call to
the statistic. Batch sizes >>1 tend to be faster when the statistic
is vectorized, but memory usage scales linearly with the batch size.
Default is ``None``, which processes all resamples in a single batch.
random_state : {None, int, `numpy.random.Generator`,
`numpy.random.RandomState`}, optional
Pseudorandom number generator state used to generate resamples.
If `random_state` is already a ``Generator`` or ``RandomState``
instance, then that instance is used.
If `random_state` is an int, a new ``RandomState`` instance is used,
seeded with `random_state`.
If `random_state` is ``None`` (default), the
`numpy.random.RandomState` singleton is used.
method : {'bca', 'percentile', 'basic'}
Whether to use the 'percentile' bootstrap ('percentile'), the 'basic'
(AKA 'reverse') bootstrap ('basic'), or the bias-corrected and
accelerated bootstrap ('BCa', default).
NrI rG r� �strr� c �p � t | j | j | j | j �� S )N)rF r< rG r� )rK rF r< rG r� rL s r% rN zBootstrapMethod._asdict4 s. � ��� 0� 0��
�
�!%�!2�!2�4�;�;�H� Hr' )r� r� r� r� rG r� r� rN r( r' r% rS rS s# � � �> �L�&� ��F�C��Hr' rS r )NN)8�
__future__r rU �numpyr � itertoolsr r r �collections.abcr r� �scipy._lib._utilr r �
scipy.specialr
r r r
r � dataclassesr �_commonr �_axis_nan_policyr r �_warnings_errorsr �__all__r0 rD rH rO r` r� r� r� r r� r� r r� r r r r% r) r- r5 r rF rH rP rS r( r' r% �<module>ra s� �� "� � � 9� 9� $� � B� 6� 6� )� !� '� G� 3�
?���,�(�P��00#�f^4�B �'� '� �'�, /3�$� �e�!�d�%�e�d� �}P�@ 9-�x �"� "� �"�$ �8�V�$�9=�!%�T�{��BF� %�BF�J �"� "� �"�$�6>�$�8 '+�99�z +/�69�t *.� 9�>;;�| ;H� $�$�d�!,�1�4�YG�x �� � ��* �#"�'� #"� �#"�L �!4�(� !4� �!4�H �&H�&� &H� �&Hr'